2024-04-28T12:18:32.121 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-28T12:18:32.121 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} --owner scheduled_rishabh@teuthology --name rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874 -- /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874/orig.config.yaml 2024-04-28T12:18:32.177 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-28T12:18:32.270 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874 branch: wip-rishabh-testing-20240426.111959 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} email: null first_in_suite: false job_id: '7676874' kernel: client: branch: testing flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240426.111959 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240426.111959 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 816 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 sleep_before_teardown: 0 subset: 166/512 suite: fs suite_branch: wip-rishabh-testing-20240426.111959 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGIgFwhf3vQnPVVn/zn96Zf863CehHmwn8t12xw3AiwW8ba02Swrvw+9m4vQ/QqLHcRw04F68ndZbDp1KGgvUj4= smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBCcIiiBvpOWfL0qT6g/oXGiHTkyqt88o58BkiWjrr94bU8jib7IsvHTiDcM9r2tYNj1WAnX0zVM5C+2gijo2mE= smithi179.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFOyXSh4ZYY5lu+FFO3Ced+cGQTgvUC82j/tH5f8Ne1HTKFH2vA8QREFAiCcMRtRaMtw4Q6j5ss4UxLXvg5qDtw= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-28_11:41:23 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-28T12:18:32.270 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa; will attempt to use it 2024-04-28T12:18:32.271 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks 2024-04-28T12:18:32.271 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-28T12:18:32.273 INFO:teuthology.task.internal:Checking packages... 2024-04-28T12:18:32.295 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc612ffb9c1d5bc27536bc917a705262a36d4387' 2024-04-28T12:18:32.295 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-28T12:18:32.295 INFO:teuthology.packaging:ref: None 2024-04-28T12:18:32.296 INFO:teuthology.packaging:tag: None 2024-04-28T12:18:32.296 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240426.111959 2024-04-28T12:18:32.296 INFO:teuthology.packaging:sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:18:32.296 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240426.111959 2024-04-28T12:18:32.517 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3306.gbc612ffb 2024-04-28T12:18:32.519 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-28T12:18:32.524 INFO:teuthology.task.internal:no buildpackages task found 2024-04-28T12:18:32.524 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-28T12:18:32.549 INFO:teuthology.task.internal:Saving configuration 2024-04-28T12:18:32.570 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-28T12:18:32.617 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-28T12:18:32.647 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 12:12:13.133132', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGIgFwhf3vQnPVVn/zn96Zf863CehHmwn8t12xw3AiwW8ba02Swrvw+9m4vQ/QqLHcRw04F68ndZbDp1KGgvUj4='} 2024-04-28T12:18:32.670 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 12:12:13.134531', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBCcIiiBvpOWfL0qT6g/oXGiHTkyqt88o58BkiWjrr94bU8jib7IsvHTiDcM9r2tYNj1WAnX0zVM5C+2gijo2mE='} 2024-04-28T12:18:32.690 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 12:12:13.131415', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFOyXSh4ZYY5lu+FFO3Ced+cGQTgvUC82j/tH5f8Ne1HTKFH2vA8QREFAiCcMRtRaMtw4Q6j5ss4UxLXvg5qDtw='} 2024-04-28T12:18:32.690 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-28T12:18:32.696 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-28T12:18:32.696 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-28T12:18:32.697 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-28T12:18:32.697 INFO:teuthology.run_tasks:Running task console_log... 2024-04-28T12:18:32.846 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feaa308d160>, signals=[15]) 2024-04-28T12:18:32.847 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-28T12:18:32.853 INFO:teuthology.task.internal:Opening connections... 2024-04-28T12:18:32.853 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2024-04-28T12:18:32.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:18:32.935 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2024-04-28T12:18:32.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:18:33.017 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2024-04-28T12:18:33.018 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:18:33.091 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-28T12:18:33.097 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-04-28T12:18:33.115 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-04-28T12:18:33.115 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-04-28T12:18:33.172 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream" 2024-04-28T12:18:33.172 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="9" 2024-04-28T12:18:33.172 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2024-04-28T12:18:33.172 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2024-04-28T12:18:33.172 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="9" 2024-04-28T12:18:33.172 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el9" 2024-04-28T12:18:33.172 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-28T12:18:33.173 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2024-04-28T12:18:33.173 INFO:teuthology.orchestra.run.smithi084.stdout:LOGO="fedora-logo-icon" 2024-04-28T12:18:33.173 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-28T12:18:33.173 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2024-04-28T12:18:33.173 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-28T12:18:33.173 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-28T12:18:33.173 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-28T12:18:33.174 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-04-28T12:18:33.196 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2024-04-28T12:18:33.214 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2024-04-28T12:18:33.214 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2024-04-28T12:18:33.271 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Stream" 2024-04-28T12:18:33.271 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="9" 2024-04-28T12:18:33.271 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2024-04-28T12:18:33.272 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2024-04-28T12:18:33.272 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="9" 2024-04-28T12:18:33.272 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el9" 2024-04-28T12:18:33.272 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-28T12:18:33.272 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2024-04-28T12:18:33.272 INFO:teuthology.orchestra.run.smithi161.stdout:LOGO="fedora-logo-icon" 2024-04-28T12:18:33.272 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-28T12:18:33.272 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/" 2024-04-28T12:18:33.272 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-28T12:18:33.272 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-28T12:18:33.272 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-28T12:18:33.272 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2024-04-28T12:18:33.295 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2024-04-28T12:18:33.313 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2024-04-28T12:18:33.313 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2024-04-28T12:18:33.371 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="CentOS Stream" 2024-04-28T12:18:33.371 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="9" 2024-04-28T12:18:33.371 INFO:teuthology.orchestra.run.smithi179.stdout:ID="centos" 2024-04-28T12:18:33.371 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="rhel fedora" 2024-04-28T12:18:33.371 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="9" 2024-04-28T12:18:33.371 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el9" 2024-04-28T12:18:33.371 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-28T12:18:33.371 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31" 2024-04-28T12:18:33.371 INFO:teuthology.orchestra.run.smithi179.stdout:LOGO="fedora-logo-icon" 2024-04-28T12:18:33.371 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-28T12:18:33.372 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://centos.org/" 2024-04-28T12:18:33.372 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-28T12:18:33.372 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-28T12:18:33.372 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-28T12:18:33.372 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2024-04-28T12:18:33.396 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-28T12:18:33.406 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-28T12:18:33.431 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-28T12:18:33.431 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2024-04-28T12:18:33.434 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2024-04-28T12:18:33.436 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2024-04-28T12:18:33.452 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-28T12:18:33.458 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-28T12:18:33.458 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2024-04-28T12:18:33.490 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2024-04-28T12:18:33.493 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2024-04-28T12:18:33.592 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-28T12:18:33.667 INFO:teuthology.run_tasks:Running task kernel... 2024-04-28T12:18:33.682 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-28T12:18:33.682 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-28T12:18:33.682 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-28T12:18:33.683 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-28T12:18:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:18:33.699 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-28T12:18:33.859 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-28T12:18:33.859 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-28T12:18:33.859 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-04-28T12:18:33.876 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-437.el9.x86_64 2024-04-28T12:18:33.876 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-28T12:18:33.876 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-28T12:18:33.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum upgrade -y linux-firmware 2024-04-28T12:18:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 7.6 MB/s | 8.0 MB 00:01 2024-04-28T12:18:51.585 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-04-28T12:18:56.076 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.2 MB 00:00 2024-04-28T12:18:58.443 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-04-28T12:19:03.602 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 4.2 MB/s | 21 MB 00:05 2024-04-28T12:19:08.676 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-28T12:19:10.712 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-04-28T12:19:10.713 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-04-28T12:19:10.713 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-04-28T12:19:10.823 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-28T12:19:10.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-28T12:19:10.969 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-28T12:19:10.969 DEBUG:teuthology.orchestra.run.smithi084:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-28T12:19:10.972 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi084.front.sepia.ceph.com to complete... 2024-04-28T12:23:36.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28 12:23:36 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-28T12:23:36.482 INFO:teuthology.orchestra.run.smithi084.stderr:FINISHED --2024-04-28 12:23:36-- 2024-04-28T12:23:36.483 INFO:teuthology.orchestra.run.smithi084.stderr:Total wall clock time: 4m 25s 2024-04-28T12:23:36.483 INFO:teuthology.orchestra.run.smithi084.stderr:Downloaded: 1 files, 341M in 4m 25s (1.29 MB/s) 2024-04-28T12:23:36.487 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-28T12:23:36.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-28T12:23:38.613 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying... ######################################## 2024-04-28T12:23:38.818 INFO:teuthology.orchestra.run.smithi084.stdout:Preparing... ######################################## 2024-04-28T12:23:38.934 INFO:teuthology.orchestra.run.smithi084.stdout:Updating / installing... 2024-04-28T12:23:46.901 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-28T12:24:24.067 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-28T12:24:24.068 DEBUG:teuthology.orchestra.run.smithi084:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-28T12:24:24.157 INFO:teuthology.orchestra.run.smithi084.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-28T12:24:24.157 INFO:teuthology.orchestra.run.smithi084.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-28T12:24:24.157 INFO:teuthology.orchestra.run.smithi084.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-28T12:24:24.157 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-28T12:24:24.158 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-28T12:24:24.158 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-28T12:24:24.158 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-28T12:24:24.158 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-28T12:24:24.158 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-28T12:24:24.158 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-28T12:24:24.159 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-28T12:24:24.159 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-28T12:24:24.159 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-28T12:24:24.159 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-28T12:24:24.159 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-28T12:24:24.159 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-28T12:24:24.159 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-28T12:24:24.160 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-28T12:24:24.160 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-28T12:24:24.160 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-28T12:24:24.160 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-28T12:24:24.160 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-28T12:24:24.160 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-28T12:24:24.160 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-28T12:24:24.161 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-28T12:24:24.161 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-28T12:24:24.161 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-28T12:24:24.161 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-28T12:24:24.161 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-28T12:24:24.161 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-28T12:24:24.162 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-28T12:24:24.162 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-28T12:24:24.162 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-28T12:24:24.162 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-28T12:24:24.162 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-28T12:24:24.162 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-28T12:24:24.163 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-28T12:24:24.163 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-28T12:24:24.163 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-28T12:24:24.163 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-28T12:24:24.163 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-28T12:24:24.163 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-28T12:24:24.164 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-28T12:24:24.164 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-28T12:24:24.164 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-28T12:24:24.164 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-28T12:24:24.164 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-28T12:24:24.164 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-28T12:24:24.164 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-28T12:24:24.164 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-28T12:24:24.164 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-28T12:24:24.164 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-28T12:24:24.164 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-28T12:24:24.164 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-28T12:24:24.165 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-28T12:24:24.166 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-28T12:24:24.167 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-28T12:24:24.168 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-28T12:24:24.169 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-28T12:24:24.170 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-28T12:24:24.171 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-28T12:24:24.172 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-28T12:24:24.173 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-28T12:24:24.174 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-28T12:24:24.175 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-28T12:24:24.176 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-28T12:24:24.176 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-28T12:24:24.176 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-28T12:24:24.176 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-28T12:24:24.176 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-28T12:24:24.176 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-28T12:24:24.176 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-28T12:24:24.176 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-28T12:24:24.176 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-28T12:24:24.176 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-28T12:24:24.176 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-28T12:24:24.176 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-28T12:24:24.177 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-28T12:24:24.178 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-28T12:24:24.179 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-28T12:24:24.180 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-28T12:24:24.181 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-28T12:24:24.182 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-28T12:24:24.183 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-28T12:24:24.184 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-28T12:24:24.184 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-28T12:24:24.184 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-28T12:24:24.184 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-28T12:24:24.184 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-28T12:24:24.184 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-28T12:24:24.184 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-28T12:24:24.184 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-28T12:24:24.185 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-28T12:24:24.186 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-28T12:24:24.186 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-28T12:24:24.186 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-28T12:24:24.186 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-28T12:24:24.186 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-28T12:24:24.186 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-28T12:24:24.186 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-28T12:24:24.186 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-28T12:24:24.186 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-28T12:24:24.186 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-28T12:24:24.186 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-28T12:24:24.187 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-28T12:24:24.188 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-28T12:24:24.189 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-28T12:24:24.190 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-28T12:24:24.191 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-28T12:24:24.192 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-28T12:24:24.193 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-28T12:24:24.194 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-28T12:24:24.194 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-28T12:24:24.194 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-28T12:24:24.194 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-28T12:24:24.194 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-28T12:24:24.194 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-28T12:24:24.194 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-28T12:24:24.194 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-28T12:24:24.194 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-28T12:24:24.194 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-28T12:24:24.195 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-28T12:24:24.195 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-28T12:24:24.195 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-28T12:24:24.195 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-28T12:24:24.195 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-28T12:24:24.195 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-28T12:24:24.195 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-28T12:24:24.195 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-28T12:24:24.195 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-28T12:24:24.195 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-28T12:24:24.195 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-28T12:24:24.196 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-28T12:24:24.196 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-28T12:24:24.196 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-28T12:24:24.196 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-28T12:24:24.196 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-28T12:24:24.196 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-28T12:24:24.196 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-28T12:24:24.196 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-28T12:24:24.196 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-28T12:24:24.196 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-28T12:24:24.196 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-28T12:24:24.196 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-28T12:24:24.197 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-28T12:24:24.198 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-28T12:24:24.198 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-28T12:24:24.198 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-28T12:24:24.198 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-28T12:24:24.198 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-28T12:24:24.198 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-28T12:24:24.198 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-28T12:24:24.198 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-28T12:24:24.198 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-28T12:24:24.198 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-28T12:24:24.198 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-28T12:24:24.198 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-28T12:24:24.199 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-28T12:24:24.200 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-28T12:24:24.201 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-28T12:24:24.202 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-28T12:24:24.202 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-28T12:24:24.202 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-28T12:24:24.202 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-28T12:24:24.202 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-28T12:24:24.202 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-28T12:24:24.202 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-28T12:24:24.202 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-28T12:24:24.202 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-28T12:24:24.202 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-28T12:24:24.202 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-28T12:24:24.202 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-28T12:24:24.203 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-28T12:24:24.204 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-28T12:24:24.205 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-28T12:24:24.206 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-28T12:24:24.207 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-28T12:24:24.208 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-28T12:24:24.209 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-28T12:24:24.210 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-28T12:24:24.210 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-28T12:24:24.210 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-28T12:24:24.210 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-28T12:24:24.210 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-28T12:24:24.210 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-28T12:24:24.210 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-28T12:24:24.210 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-28T12:24:24.210 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-28T12:24:24.210 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-28T12:24:24.210 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-28T12:24:24.211 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-28T12:24:24.212 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-28T12:24:24.213 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-28T12:24:24.214 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-28T12:24:24.215 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-28T12:24:24.216 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-28T12:24:24.217 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-28T12:24:24.218 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-28T12:24:24.218 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-28T12:24:24.218 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-28T12:24:24.218 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-28T12:24:24.218 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-28T12:24:24.218 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-28T12:24:24.218 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-28T12:24:24.218 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-28T12:24:24.218 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-28T12:24:24.218 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-28T12:24:24.218 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-28T12:24:24.219 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-28T12:24:24.219 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-28T12:24:24.219 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-28T12:24:24.219 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-28T12:24:24.219 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-28T12:24:24.219 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-28T12:24:24.219 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-28T12:24:24.219 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-28T12:24:24.219 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-28T12:24:24.219 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-28T12:24:24.219 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-28T12:24:24.219 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-28T12:24:24.220 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-28T12:24:24.221 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-28T12:24:24.221 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-28T12:24:24.221 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-28T12:24:24.221 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-28T12:24:24.221 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-28T12:24:24.221 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-28T12:24:24.221 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-28T12:24:24.221 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-28T12:24:24.221 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-28T12:24:24.221 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-28T12:24:24.221 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-28T12:24:24.221 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-28T12:24:24.222 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-28T12:24:24.222 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-28T12:24:24.222 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-28T12:24:24.222 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-28T12:24:24.222 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-28T12:24:24.222 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-28T12:24:24.222 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-28T12:24:24.222 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-28T12:24:24.222 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-28T12:24:24.222 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-28T12:24:24.222 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-28T12:24:24.222 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-28T12:24:24.223 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-28T12:24:24.224 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-28T12:24:24.225 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-28T12:24:24.226 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-28T12:24:24.227 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-28T12:24:24.228 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-28T12:24:24.229 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-28T12:24:24.230 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-28T12:24:24.230 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-28T12:24:24.230 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-28T12:24:24.230 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-28T12:24:24.230 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-28T12:24:24.230 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-28T12:24:24.230 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-28T12:24:24.230 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-28T12:24:24.230 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-28T12:24:24.230 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-28T12:24:24.231 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-28T12:24:24.231 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-28T12:24:24.231 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-28T12:24:24.231 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-28T12:24:24.231 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-28T12:24:24.231 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-28T12:24:24.231 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-28T12:24:24.231 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-28T12:24:24.231 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-28T12:24:24.231 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-28T12:24:24.231 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-28T12:24:24.232 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-28T12:24:24.233 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-28T12:24:24.234 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-28T12:24:24.234 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-28T12:24:24.234 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-28T12:24:24.234 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-28T12:24:24.234 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-28T12:24:24.234 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-28T12:24:24.234 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-28T12:24:24.234 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-28T12:24:24.234 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-28T12:24:24.234 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-28T12:24:24.234 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-28T12:24:24.234 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-28T12:24:24.235 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-28T12:24:24.236 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-28T12:24:24.237 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-28T12:24:24.238 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-28T12:24:24.238 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-28T12:24:24.238 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-28T12:24:24.238 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-28T12:24:24.238 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-28T12:24:24.238 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-28T12:24:24.238 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-28T12:24:24.238 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-28T12:24:24.238 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-28T12:24:24.238 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-28T12:24:24.238 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-28T12:24:24.238 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-28T12:24:24.239 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-28T12:24:24.240 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-28T12:24:24.241 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-28T12:24:24.242 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-28T12:24:24.243 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-28T12:24:24.243 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-28T12:24:24.243 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-28T12:24:24.243 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-28T12:24:24.243 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-28T12:24:24.243 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-28T12:24:24.243 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-28T12:24:24.243 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-28T12:24:24.243 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-28T12:24:24.243 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-28T12:24:24.243 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-28T12:24:24.243 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-28T12:24:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-28T12:24:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-28T12:24:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-28T12:24:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-28T12:24:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-28T12:24:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-28T12:24:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-28T12:24:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-28T12:24:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-28T12:24:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-28T12:24:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-28T12:24:24.244 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-28T12:24:24.245 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-28T12:24:24.245 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-28T12:24:24.245 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-28T12:24:24.245 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-28T12:24:24.245 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-28T12:24:24.245 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-28T12:24:24.245 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-28T12:24:24.245 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-28T12:24:24.246 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-28T12:24:24.246 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-28T12:24:24.246 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-28T12:24:24.246 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-28T12:24:24.246 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-28T12:24:24.246 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-28T12:24:24.247 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-28T12:24:24.247 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-28T12:24:24.247 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-28T12:24:24.247 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-28T12:24:24.247 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-28T12:24:24.247 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-28T12:24:24.248 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-28T12:24:24.248 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-28T12:24:24.248 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-28T12:24:24.248 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-28T12:24:24.248 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-28T12:24:24.248 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-28T12:24:24.248 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-28T12:24:24.249 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-28T12:24:24.249 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-28T12:24:24.249 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-28T12:24:24.249 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-28T12:24:24.249 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-28T12:24:24.249 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-28T12:24:24.249 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-28T12:24:24.250 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-28T12:24:24.250 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-28T12:24:24.250 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-28T12:24:24.250 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-28T12:24:24.250 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-28T12:24:24.250 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-28T12:24:24.251 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-28T12:24:24.251 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-28T12:24:24.251 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-28T12:24:24.251 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-28T12:24:24.251 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-28T12:24:24.251 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-28T12:24:24.251 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-28T12:24:24.252 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-28T12:24:24.252 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-28T12:24:24.252 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-28T12:24:24.252 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-28T12:24:24.252 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-28T12:24:24.252 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-28T12:24:24.253 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-28T12:24:24.253 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-28T12:24:24.253 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-28T12:24:24.253 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-28T12:24:24.253 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-28T12:24:24.253 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-28T12:24:24.253 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-28T12:24:24.254 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-28T12:24:24.254 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-28T12:24:24.254 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-28T12:24:24.254 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-28T12:24:24.254 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-28T12:24:24.254 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-28T12:24:24.254 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-28T12:24:24.255 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-28T12:24:24.255 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-28T12:24:24.255 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-28T12:24:24.255 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-28T12:24:24.255 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-28T12:24:24.255 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-28T12:24:24.255 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-28T12:24:24.256 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-28T12:24:24.256 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-28T12:24:24.256 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-28T12:24:24.256 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-28T12:24:24.256 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-28T12:24:24.257 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-28T12:24:24.257 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-28T12:24:24.257 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-28T12:24:24.257 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-28T12:24:24.257 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-28T12:24:24.257 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-28T12:24:24.258 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-28T12:24:24.258 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-28T12:24:24.258 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-28T12:24:24.258 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-28T12:24:24.258 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-28T12:24:24.258 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-28T12:24:24.258 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-28T12:24:24.258 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-28T12:24:24.258 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-28T12:24:24.259 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-28T12:24:24.259 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-28T12:24:24.259 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-28T12:24:24.259 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-28T12:24:24.259 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-28T12:24:24.259 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-28T12:24:24.259 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-28T12:24:24.259 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-28T12:24:24.260 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-28T12:24:24.260 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-28T12:24:24.260 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-28T12:24:24.260 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-28T12:24:24.260 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-28T12:24:24.260 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-28T12:24:24.260 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-28T12:24:24.260 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-28T12:24:24.261 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-28T12:24:24.261 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-28T12:24:24.261 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-28T12:24:24.261 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-28T12:24:24.261 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-28T12:24:24.261 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-28T12:24:24.261 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-28T12:24:24.261 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-28T12:24:24.262 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-28T12:24:24.262 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-28T12:24:24.262 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-28T12:24:24.262 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-28T12:24:24.262 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-28T12:24:24.262 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-28T12:24:24.262 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-28T12:24:24.262 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-28T12:24:24.262 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-28T12:24:24.263 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-28T12:24:24.263 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-28T12:24:24.263 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-28T12:24:24.263 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-28T12:24:24.263 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-28T12:24:24.263 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-28T12:24:24.263 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-28T12:24:24.263 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-28T12:24:24.264 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-28T12:24:24.264 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-28T12:24:24.264 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-28T12:24:24.264 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-28T12:24:24.264 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-28T12:24:24.264 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-28T12:24:24.264 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-28T12:24:24.264 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-28T12:24:24.265 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-28T12:24:24.265 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-28T12:24:24.265 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-28T12:24:24.265 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-28T12:24:24.265 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-28T12:24:24.265 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-28T12:24:24.265 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-28T12:24:24.265 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-28T12:24:24.265 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-28T12:24:24.266 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-28T12:24:24.266 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-28T12:24:24.266 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-28T12:24:24.266 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-28T12:24:24.266 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-28T12:24:24.266 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-28T12:24:24.266 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-28T12:24:24.266 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-28T12:24:24.267 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-28T12:24:24.267 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-28T12:24:24.267 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-28T12:24:24.267 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-28T12:24:24.267 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-28T12:24:24.267 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-28T12:24:24.267 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-28T12:24:24.267 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-28T12:24:24.268 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-28T12:24:24.268 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-28T12:24:24.268 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-28T12:24:24.268 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-28T12:24:24.268 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-28T12:24:24.268 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-28T12:24:24.268 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-28T12:24:24.268 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-28T12:24:24.268 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-28T12:24:24.269 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-28T12:24:24.269 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-28T12:24:24.269 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-28T12:24:24.269 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-28T12:24:24.269 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-28T12:24:24.269 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-28T12:24:24.269 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-28T12:24:24.269 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-28T12:24:24.269 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-28T12:24:24.269 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-28T12:24:24.270 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-28T12:24:24.270 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-28T12:24:24.270 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-28T12:24:24.270 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-28T12:24:24.270 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-28T12:24:24.270 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-28T12:24:24.270 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-28T12:24:24.270 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-28T12:24:24.271 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-28T12:24:24.271 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-28T12:24:24.271 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-28T12:24:24.271 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-28T12:24:24.271 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-28T12:24:24.271 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-28T12:24:24.271 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-28T12:24:24.271 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-28T12:24:24.271 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-28T12:24:24.272 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-28T12:24:24.272 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-28T12:24:24.272 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-28T12:24:24.272 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-28T12:24:24.272 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-28T12:24:24.272 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-28T12:24:24.272 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-28T12:24:24.272 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-28T12:24:24.273 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-28T12:24:24.274 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-28T12:24:24.274 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-28T12:24:24.274 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-28T12:24:24.274 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-28T12:24:24.274 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-28T12:24:24.274 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-28T12:24:24.274 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-28T12:24:24.274 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-28T12:24:24.274 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-28T12:24:24.274 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-28T12:24:24.274 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-28T12:24:24.274 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-28T12:24:24.275 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-28T12:24:24.276 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-28T12:24:24.276 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-28T12:24:24.276 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-28T12:24:24.276 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-28T12:24:24.276 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-28T12:24:24.276 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-28T12:24:24.276 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-28T12:24:24.276 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-28T12:24:24.276 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-28T12:24:24.276 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-28T12:24:24.276 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-28T12:24:24.276 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-28T12:24:24.277 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-28T12:24:24.278 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-28T12:24:24.279 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-28T12:24:24.280 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-28T12:24:24.281 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-28T12:24:24.282 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-28T12:24:24.283 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-28T12:24:24.284 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-28T12:24:24.285 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-28T12:24:24.286 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-28T12:24:24.287 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-28T12:24:24.288 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-28T12:24:24.289 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-28T12:24:24.289 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-28T12:24:24.289 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-28T12:24:24.289 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-28T12:24:24.289 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-28T12:24:24.289 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-28T12:24:24.290 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-28T12:24:24.290 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-28T12:24:24.290 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-28T12:24:24.290 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-28T12:24:24.290 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-28T12:24:24.290 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-28T12:24:24.290 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-28T12:24:24.290 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-28T12:24:24.290 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-28T12:24:24.290 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-28T12:24:24.290 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-28T12:24:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-28T12:24:24.292 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-28T12:24:24.293 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-28T12:24:24.294 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-28T12:24:24.294 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-28T12:24:24.294 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-28T12:24:24.294 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-28T12:24:24.294 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-28T12:24:24.294 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-28T12:24:24.294 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-28T12:24:24.294 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-28T12:24:24.294 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-28T12:24:24.294 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-28T12:24:24.295 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-28T12:24:24.296 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-28T12:24:24.297 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-28T12:24:24.298 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-28T12:24:24.299 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-28T12:24:24.300 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-28T12:24:24.301 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-28T12:24:24.302 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-28T12:24:24.303 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-28T12:24:24.304 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-28T12:24:24.305 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-28T12:24:24.306 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-28T12:24:24.307 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-28T12:24:24.308 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-28T12:24:24.309 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-28T12:24:24.310 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-28T12:24:24.311 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-28T12:24:24.312 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-28T12:24:24.313 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-28T12:24:24.314 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-28T12:24:24.315 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-28T12:24:24.315 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-28T12:24:24.315 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-28T12:24:24.315 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-28T12:24:24.315 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-28T12:24:24.315 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-28T12:24:24.315 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-28T12:24:24.315 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-28T12:24:24.316 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-28T12:24:24.316 INFO:teuthology.orchestra.run.smithi084.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-28T12:24:24.323 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-28T12:24:24.324 DEBUG:teuthology.orchestra.run.smithi084:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-28T12:24:24.346 INFO:teuthology.orchestra.run.smithi084.stdout:postinstall scriptlet (using /bin/sh): 2024-04-28T12:24:24.346 INFO:teuthology.orchestra.run.smithi084.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout:fi 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout:for file in vmlinuz System.map config; do 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout: fi 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout:done 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout:fi 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-28T12:24:24.347 INFO:teuthology.orchestra.run.smithi084.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-28T12:24:24.348 INFO:teuthology.orchestra.run.smithi084.stdout:fi 2024-04-28T12:24:24.348 INFO:teuthology.orchestra.run.smithi084.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-28T12:24:24.348 INFO:teuthology.orchestra.run.smithi084.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-28T12:24:24.348 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-28T12:24:24.348 INFO:teuthology.orchestra.run.smithi084.stdout:fi 2024-04-28T12:24:24.349 DEBUG:teuthology.orchestra.run.smithi084:> sudo rpm -qi grub2-tools 2024-04-28T12:24:24.419 INFO:teuthology.orchestra.run.smithi084.stdout:Name : grub2-tools 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Epoch : 1 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Version : 2.06 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Release : 68.el9 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: x86_64 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Group : Unspecified 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Size : 8264254 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:License : GPLv3+ 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Packager : builder@centos.org 2024-04-28T12:24:24.420 INFO:teuthology.orchestra.run.smithi084.stdout:Vendor : CentOS 2024-04-28T12:24:24.421 INFO:teuthology.orchestra.run.smithi084.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-28T12:24:24.421 INFO:teuthology.orchestra.run.smithi084.stdout:Summary : Support tools for GRUB. 2024-04-28T12:24:24.421 INFO:teuthology.orchestra.run.smithi084.stdout:Description : 2024-04-28T12:24:24.421 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:24:24.421 INFO:teuthology.orchestra.run.smithi084.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-28T12:24:24.421 INFO:teuthology.orchestra.run.smithi084.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-28T12:24:24.421 INFO:teuthology.orchestra.run.smithi084.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-28T12:24:24.421 INFO:teuthology.orchestra.run.smithi084.stdout:hardware devices. 2024-04-28T12:24:24.421 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:24:24.421 INFO:teuthology.orchestra.run.smithi084.stdout:This subpackage provides tools for support of all platforms. 2024-04-28T12:24:24.422 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-28T12:24:24.422 INFO:teuthology.task.kernel:Updating grub on smithi084 to boot 6.9.0-rc3-g806ec36b2112 2024-04-28T12:24:24.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-28T12:24:24.951 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2024-04-28T12:24:26.299 INFO:teuthology.orchestra.run.smithi084.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-28T12:24:26.327 INFO:teuthology.orchestra.run.smithi084.stderr:done 2024-04-28T12:24:26.330 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-04-28T12:24:26.345 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.OQFz8lsCYn 2024-04-28T12:24:26.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OQFz8lsCYn 2024-04-28T12:24:26.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.OQFz8lsCYn 2024-04-28T12:24:26.560 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.OQFz8lsCYn is 6KB 2024-04-28T12:24:26.580 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.OQFz8lsCYn 2024-04-28T12:24:26.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo /bin/ls /boot/loader/entries || true 2024-04-28T12:24:26.671 INFO:teuthology.orchestra.run.smithi084.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-28T12:24:26.671 INFO:teuthology.orchestra.run.smithi084.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-28T12:24:26.671 INFO:teuthology.orchestra.run.smithi084.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-28T12:24:26.672 INFO:teuthology.orchestra.run.smithi084.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-28T12:24:26.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-28T12:24:26.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2024-04-28T12:24:26.921 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-04-28T12:24:56.827 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-28T12:24:56.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-28T12:24:56.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:25:56.831 DEBUG:teuthology.orchestra.remote:timed out 2024-04-28T12:26:05.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-28T12:26:05.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:26:09.137 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-04-28T12:26:09.759 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-28T12:26:09.760 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-28T12:26:09.760 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-04-28T12:26:09.803 INFO:teuthology.orchestra.run.smithi084.stdout:6.9.0-rc3-g806ec36b2112 2024-04-28T12:26:09.803 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-28T12:26:09.803 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-28T12:26:09.804 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-28T12:26:10.807 DEBUG:teuthology.parallel:result is None 2024-04-28T12:26:10.808 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-28T12:26:10.818 INFO:teuthology.task.internal:Creating test directory... 2024-04-28T12:26:10.818 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-28T12:26:10.826 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-28T12:26:10.829 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-28T12:26:10.872 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-28T12:26:10.879 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-28T12:26:10.885 INFO:teuthology.task.internal:Creating archive directory... 2024-04-28T12:26:10.886 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-28T12:26:10.891 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-28T12:26:10.893 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-28T12:26:11.141 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-28T12:26:11.149 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-28T12:26:11.149 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-28T12:26:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:26:11.167 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-28T12:26:11.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:26:11.184 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-28T12:26:11.189 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-28T12:26:11.210 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-28T12:26:11.249 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T12:26:11.268 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T12:26:11.270 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T12:26:11.288 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T12:26:11.304 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T12:26:11.351 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-28T12:26:11.355 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-28T12:26:11.362 INFO:teuthology.task.internal:Configuring sudo... 2024-04-28T12:26:11.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-28T12:26:11.369 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-28T12:26:11.372 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-28T12:26:11.437 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-28T12:26:11.448 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-28T12:26:11.451 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-28T12:26:11.456 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-28T12:26:11.458 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-28T12:26:11.496 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T12:26:11.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T12:26:11.714 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T12:26:11.793 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T12:26:11.863 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:26:11.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-28T12:26:11.957 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T12:26:12.003 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T12:26:12.055 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T12:26:12.135 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T12:26:12.179 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-28T12:26:12.179 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-28T12:26:12.244 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T12:26:12.287 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-28T12:26:12.347 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T12:26:12.433 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-28T12:26:12.473 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-28T12:26:12.473 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-28T12:26:12.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2024-04-28T12:26:12.547 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2024-04-28T12:26:12.549 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2024-04-28T12:26:12.578 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T12:26:12.613 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T12:26:12.659 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T12:26:13.031 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-28T12:26:13.040 INFO:teuthology.task.internal:Starting timer... 2024-04-28T12:26:13.040 INFO:teuthology.run_tasks:Running task pcp... 2024-04-28T12:26:13.051 INFO:teuthology.run_tasks:Running task selinux... 2024-04-28T12:26:13.063 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-28T12:26:13.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo service auditd rotate 2024-04-28T12:26:13.175 INFO:teuthology.orchestra.run.smithi084.stdout:Rotating logs: 2024-04-28T12:26:13.180 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2024-04-28T12:26:13.249 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2024-04-28T12:26:13.251 DEBUG:teuthology.orchestra.run.smithi179:> sudo service auditd rotate 2024-04-28T12:26:13.317 INFO:teuthology.orchestra.run.smithi179.stdout:Rotating logs: 2024-04-28T12:26:13.319 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-28T12:26:13.319 DEBUG:teuthology.orchestra.run.smithi084:> /usr/sbin/getenforce 2024-04-28T12:26:13.371 INFO:teuthology.orchestra.run.smithi084.stdout:Permissive 2024-04-28T12:26:13.372 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2024-04-28T12:26:13.415 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2024-04-28T12:26:13.415 DEBUG:teuthology.orchestra.run.smithi179:> /usr/sbin/getenforce 2024-04-28T12:26:13.438 INFO:teuthology.orchestra.run.smithi179.stdout:Permissive 2024-04-28T12:26:13.439 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi084.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi161.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi179.front.sepia.ceph.com': 'permissive'} 2024-04-28T12:26:13.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T12:26:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:26:13.507 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T12:26:13.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:26:13.535 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T12:26:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T12:26:13.563 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-28T12:26:13.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo /usr/sbin/setenforce permissive 2024-04-28T12:26:13.759 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2024-04-28T12:26:13.785 DEBUG:teuthology.orchestra.run.smithi179:> sudo /usr/sbin/setenforce permissive 2024-04-28T12:26:13.811 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-28T12:26:13.821 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-28T12:26:13.840 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-28T12:26:13.842 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi084.front.sepia.ceph.com,smithi161.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-28T12:31:18.902 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi084.front.sepia.ceph.com'), Remote(name='ubuntu@smithi161.front.sepia.ceph.com'), Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2024-04-28T12:31:18.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-28T12:31:18.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:31:19.014 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-04-28T12:31:19.165 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-28T12:31:19.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-28T12:31:19.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:31:19.252 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-04-28T12:31:19.330 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-28T12:31:19.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-04-28T12:31:19.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-28T12:31:19.407 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-04-28T12:31:19.495 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-04-28T12:31:19.496 INFO:teuthology.run_tasks:Running task clock... 2024-04-28T12:31:19.507 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-28T12:31:19.507 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-28T12:31:19.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T12:31:19.515 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-28T12:31:19.516 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T12:31:19.518 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-28T12:31:19.518 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T12:31:19.546 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-28T12:31:19.561 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-28T12:31:19.568 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-28T12:31:19.584 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-28T12:31:19.586 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: ntpd: command not found 2024-04-28T12:31:19.597 INFO:teuthology.orchestra.run.smithi161.stdout:506 Cannot talk to daemon 2024-04-28T12:31:19.601 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-28T12:31:19.608 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: ntpd: command not found 2024-04-28T12:31:19.610 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-28T12:31:19.619 INFO:teuthology.orchestra.run.smithi179.stdout:506 Cannot talk to daemon 2024-04-28T12:31:19.624 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-28T12:31:19.634 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-28T12:31:19.641 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-28T12:31:19.647 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-28T12:31:19.678 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-04-28T12:31:19.682 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T12:31:19.682 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-04-28T12:31:19.682 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:31:19.682 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:31:19.682 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:31:19.682 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:31:19.707 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-04-28T12:31:19.710 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T12:31:19.711 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-04-28T12:31:19.711 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:31:19.711 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:31:19.711 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:31:19.711 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:31:19.713 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: ntpd: command not found 2024-04-28T12:31:19.744 INFO:teuthology.orchestra.run.smithi084.stdout:506 Cannot talk to daemon 2024-04-28T12:31:19.786 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-28T12:31:19.826 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-28T12:31:19.947 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2024-04-28T12:31:19.956 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T12:31:19.956 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-04-28T12:31:19.956 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:31:19.956 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:31:19.956 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:31:19.957 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T12:31:19.958 INFO:teuthology.run_tasks:Running task install... 2024-04-28T12:31:19.966 DEBUG:teuthology.task.install:project ceph 2024-04-28T12:31:19.967 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}} 2024-04-28T12:31:19.967 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'} 2024-04-28T12:31:19.967 INFO:teuthology.task.install:Using flavor: default 2024-04-28T12:31:19.975 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-28T12:31:19.975 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-28T12:31:19.976 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-28T12:31:19.976 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:31:19.981 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-28T12:31:19.981 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:31:19.983 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-28T12:31:19.983 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:31:20.123 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-28T12:31:20.123 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-28T12:31:20.202 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-28T12:31:20.202 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-28T12:31:20.214 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-28T12:31:20.214 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-28T12:31:20.464 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-28T12:31:20.464 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-28T12:31:20.464 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-28T12:31:20.491 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-28T12:31:20.492 DEBUG:teuthology.orchestra.run.smithi161:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-28T12:31:20.557 DEBUG:teuthology.orchestra.run.smithi161:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-28T12:31:20.574 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-28T12:31:20.574 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-28T12:31:20.574 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-28T12:31:20.603 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-28T12:31:20.603 DEBUG:teuthology.orchestra.run.smithi179:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-28T12:31:20.611 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-28T12:31:20.611 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:31:20.611 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-28T12:31:20.632 DEBUG:teuthology.orchestra.run.smithi161:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-28T12:31:20.671 DEBUG:teuthology.orchestra.run.smithi179:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-28T12:31:20.696 INFO:teuthology.orchestra.run.smithi161.stdout:check_obsoletes = 1 2024-04-28T12:31:20.697 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum clean all 2024-04-28T12:31:20.702 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-28T12:31:20.702 DEBUG:teuthology.orchestra.run.smithi084:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-28T12:31:20.749 DEBUG:teuthology.orchestra.run.smithi179:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-28T12:31:20.814 INFO:teuthology.orchestra.run.smithi179.stdout:check_obsoletes = 1 2024-04-28T12:31:20.816 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum clean all 2024-04-28T12:31:20.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-28T12:31:20.923 INFO:teuthology.orchestra.run.smithi161.stdout:48 files removed 2024-04-28T12:31:20.942 DEBUG:teuthology.orchestra.run.smithi084:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-28T12:31:20.950 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-28T12:31:21.021 INFO:teuthology.orchestra.run.smithi084.stdout:check_obsoletes = 1 2024-04-28T12:31:21.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum clean all 2024-04-28T12:31:21.043 INFO:teuthology.orchestra.run.smithi179.stdout:48 files removed 2024-04-28T12:31:21.070 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-28T12:31:21.352 INFO:teuthology.orchestra.run.smithi084.stdout:48 files removed 2024-04-28T12:31:21.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-28T12:31:22.186 INFO:teuthology.orchestra.run.smithi161.stdout:ceph packages for x86_64 86 kB/s | 84 kB 00:00 2024-04-28T12:31:22.233 INFO:teuthology.orchestra.run.smithi179.stdout:ceph packages for x86_64 93 kB/s | 84 kB 00:00 2024-04-28T12:31:22.860 INFO:teuthology.orchestra.run.smithi084.stdout:ceph packages for x86_64 81 kB/s | 84 kB 00:01 2024-04-28T12:31:23.112 INFO:teuthology.orchestra.run.smithi161.stdout:ceph noarch packages 13 kB/s | 12 kB 00:00 2024-04-28T12:31:23.125 INFO:teuthology.orchestra.run.smithi179.stdout:ceph noarch packages 14 kB/s | 12 kB 00:00 2024-04-28T12:31:23.777 INFO:teuthology.orchestra.run.smithi084.stdout:ceph noarch packages 14 kB/s | 12 kB 00:00 2024-04-28T12:31:23.795 INFO:teuthology.orchestra.run.smithi161.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-04-28T12:31:23.884 INFO:teuthology.orchestra.run.smithi179.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-04-28T12:31:24.401 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-04-28T12:31:24.545 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-28T12:31:24.661 INFO:teuthology.orchestra.run.smithi084.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-04-28T12:31:26.217 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-04-28T12:31:27.032 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-28T12:31:31.772 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 6.2 MB 00:00 2024-04-28T12:31:33.215 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-04-28T12:31:34.569 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-28T12:31:39.583 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-28T12:31:41.695 INFO:teuthology.orchestra.run.smithi179.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T12:31:41.696 INFO:teuthology.orchestra.run.smithi179.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T12:31:41.711 INFO:teuthology.orchestra.run.smithi179.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-28T12:31:41.711 INFO:teuthology.orchestra.run.smithi179.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-28T12:31:41.711 INFO:teuthology.orchestra.run.smithi179.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-28T12:31:41.711 INFO:teuthology.orchestra.run.smithi179.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-28T12:31:41.711 INFO:teuthology.orchestra.run.smithi179.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-28T12:31:41.713 INFO:teuthology.orchestra.run.smithi179.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-28T12:31:41.713 INFO:teuthology.orchestra.run.smithi179.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-28T12:31:41.772 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-04-28T12:31:41.782 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================= 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================= 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-28T12:31:41.783 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-28T12:31:41.784 INFO:teuthology.orchestra.run.smithi179.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-28T12:31:41.785 INFO:teuthology.orchestra.run.smithi179.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrading: 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-28T12:31:41.786 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-28T12:31:41.787 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-28T12:31:41.788 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-28T12:31:41.789 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-28T12:31:41.790 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-28T12:31:41.791 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-28T12:31:41.792 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-28T12:31:41.793 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-28T12:31:41.794 INFO:teuthology.orchestra.run.smithi179.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-28T12:31:41.794 INFO:teuthology.orchestra.run.smithi179.stdout:Installing weak dependencies: 2024-04-28T12:31:41.794 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-28T12:31:41.794 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:31:41.794 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-04-28T12:31:41.794 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================= 2024-04-28T12:31:41.794 INFO:teuthology.orchestra.run.smithi179.stdout:Install 148 Packages 2024-04-28T12:31:41.794 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrade 4 Packages 2024-04-28T12:31:41.794 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:31:41.795 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 222 M 2024-04-28T12:31:41.795 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-04-28T12:31:42.960 INFO:teuthology.orchestra.run.smithi179.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-04-28T12:31:44.193 INFO:teuthology.orchestra.run.smithi179.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 721 kB/s | 889 kB 00:01 2024-04-28T12:31:44.427 INFO:teuthology.orchestra.run.smithi179.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 622 kB/s | 145 kB 00:00 2024-04-28T12:31:46.244 INFO:teuthology.orchestra.run.smithi179.stdout:(4/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-04-28T12:31:46.727 INFO:teuthology.orchestra.run.smithi179.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 3.1 MB/s | 1.5 MB 00:00 2024-04-28T12:31:46.928 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 914 kB/s | 19 MB 00:21 2024-04-28T12:31:46.936 INFO:teuthology.orchestra.run.smithi179.stdout:(6/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 964 kB/s | 2.4 MB 00:02 2024-04-28T12:31:49.132 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 897 kB/s | 19 MB 00:21 2024-04-28T12:31:50.928 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-04-28T12:31:51.136 INFO:teuthology.orchestra.run.smithi179.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 1.1 MB/s | 4.7 MB 00:04 2024-04-28T12:31:52.545 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-28T12:31:54.090 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 5.6 MB/s | 6.2 MB 00:01 2024-04-28T12:31:55.832 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 21 MB 00:03 2024-04-28T12:31:55.882 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-28T12:31:57.142 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-28T12:32:00.796 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-28T12:32:02.896 INFO:teuthology.orchestra.run.smithi161.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T12:32:02.897 INFO:teuthology.orchestra.run.smithi161.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T12:32:02.907 INFO:teuthology.orchestra.run.smithi161.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-28T12:32:02.908 INFO:teuthology.orchestra.run.smithi161.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-28T12:32:02.909 INFO:teuthology.orchestra.run.smithi161.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-28T12:32:02.910 INFO:teuthology.orchestra.run.smithi161.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-28T12:32:02.911 INFO:teuthology.orchestra.run.smithi161.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-28T12:32:02.913 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-28T12:32:02.914 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-28T12:32:02.976 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-04-28T12:32:02.992 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-28T12:32:03.001 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================= 2024-04-28T12:32:03.001 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-04-28T12:32:03.002 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================= 2024-04-28T12:32:03.002 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-04-28T12:32:03.002 INFO:teuthology.orchestra.run.smithi161.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-28T12:32:03.002 INFO:teuthology.orchestra.run.smithi161.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-28T12:32:03.002 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T12:32:03.002 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-28T12:32:03.002 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-28T12:32:03.002 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-28T12:32:03.002 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-28T12:32:03.002 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-28T12:32:03.002 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-28T12:32:03.002 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-28T12:32:03.003 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-28T12:32:03.004 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-28T12:32:03.005 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-28T12:32:03.006 INFO:teuthology.orchestra.run.smithi161.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-28T12:32:03.007 INFO:teuthology.orchestra.run.smithi161.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-28T12:32:03.008 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-28T12:32:03.009 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-28T12:32:03.010 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-28T12:32:03.011 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-28T12:32:03.012 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout:Installing weak dependencies: 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================= 2024-04-28T12:32:03.013 INFO:teuthology.orchestra.run.smithi161.stdout:Install 148 Packages 2024-04-28T12:32:03.014 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 4 Packages 2024-04-28T12:32:03.014 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:32:03.014 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 222 M 2024-04-28T12:32:03.014 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-04-28T12:32:04.189 INFO:teuthology.orchestra.run.smithi161.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-04-28T12:32:05.348 INFO:teuthology.orchestra.run.smithi084.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T12:32:05.350 INFO:teuthology.orchestra.run.smithi084.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-28T12:32:05.369 INFO:teuthology.orchestra.run.smithi084.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-28T12:32:05.371 INFO:teuthology.orchestra.run.smithi084.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-28T12:32:05.372 INFO:teuthology.orchestra.run.smithi084.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-28T12:32:05.373 INFO:teuthology.orchestra.run.smithi084.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-28T12:32:05.375 INFO:teuthology.orchestra.run.smithi084.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-28T12:32:05.379 INFO:teuthology.orchestra.run.smithi084.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-28T12:32:05.380 INFO:teuthology.orchestra.run.smithi084.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-28T12:32:05.449 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-04-28T12:32:05.475 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================= 2024-04-28T12:32:05.475 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-04-28T12:32:05.475 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================= 2024-04-28T12:32:05.475 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-04-28T12:32:05.475 INFO:teuthology.orchestra.run.smithi084.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-28T12:32:05.475 INFO:teuthology.orchestra.run.smithi084.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-28T12:32:05.475 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T12:32:05.475 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-28T12:32:05.475 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-28T12:32:05.475 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-28T12:32:05.475 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-28T12:32:05.476 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T12:32:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrading: 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-28T12:32:05.478 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-28T12:32:05.479 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-28T12:32:05.480 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-28T12:32:05.481 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-28T12:32:05.482 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-28T12:32:05.483 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-28T12:32:05.484 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-28T12:32:05.485 INFO:teuthology.orchestra.run.smithi084.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-28T12:32:05.486 INFO:teuthology.orchestra.run.smithi084.stdout:Installing weak dependencies: 2024-04-28T12:32:05.486 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-28T12:32:05.486 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:32:05.486 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-04-28T12:32:05.486 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================= 2024-04-28T12:32:05.486 INFO:teuthology.orchestra.run.smithi084.stdout:Install 148 Packages 2024-04-28T12:32:05.486 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrade 4 Packages 2024-04-28T12:32:05.486 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:32:05.487 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 222 M 2024-04-28T12:32:05.487 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-04-28T12:32:05.656 INFO:teuthology.orchestra.run.smithi161.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 606 kB/s | 889 kB 00:01 2024-04-28T12:32:06.181 INFO:teuthology.orchestra.run.smithi161.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 277 kB/s | 145 kB 00:00 2024-04-28T12:32:06.593 INFO:teuthology.orchestra.run.smithi084.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-04-28T12:32:06.944 INFO:teuthology.orchestra.run.smithi179.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 686 kB/s | 11 MB 00:15 2024-04-28T12:32:07.061 INFO:teuthology.orchestra.run.smithi179.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 216 kB/s | 25 kB 00:00 2024-04-28T12:32:09.559 INFO:teuthology.orchestra.run.smithi084.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 300 kB/s | 889 kB 00:02 2024-04-28T12:32:10.226 INFO:teuthology.orchestra.run.smithi084.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 218 kB/s | 145 kB 00:00 2024-04-28T12:32:10.418 INFO:teuthology.orchestra.run.smithi084.stdout:(4/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 1.3 MB/s | 5.4 MB 00:04 2024-04-28T12:32:12.364 INFO:teuthology.orchestra.run.smithi161.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 391 kB/s | 2.4 MB 00:06 2024-04-28T12:32:13.243 INFO:teuthology.orchestra.run.smithi084.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 549 kB/s | 1.5 MB 00:02 2024-04-28T12:32:13.272 INFO:teuthology.orchestra.run.smithi161.stdout:(5/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 582 kB/s | 5.4 MB 00:09 2024-04-28T12:32:16.392 INFO:teuthology.orchestra.run.smithi084.stdout:(6/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 392 kB/s | 2.4 MB 00:06 2024-04-28T12:32:16.683 INFO:teuthology.orchestra.run.smithi161.stdout:(6/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 359 kB/s | 1.5 MB 00:04 2024-04-28T12:32:19.877 INFO:teuthology.orchestra.run.smithi179.stdout:(10/152): ceph-common-19.0.0-3306.gbc612ffb.el9 581 kB/s | 21 MB 00:37 2024-04-28T12:32:21.077 INFO:teuthology.orchestra.run.smithi179.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 187 kB/s | 225 kB 00:01 2024-04-28T12:32:21.285 INFO:teuthology.orchestra.run.smithi179.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 130 kB/s | 27 kB 00:00 2024-04-28T12:32:21.494 INFO:teuthology.orchestra.run.smithi179.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 152 kB/s | 32 kB 00:00 2024-04-28T12:32:22.750 INFO:teuthology.orchestra.run.smithi084.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 506 kB/s | 4.7 MB 00:09 2024-04-28T12:32:24.236 INFO:teuthology.orchestra.run.smithi179.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 272 kB/s | 747 kB 00:02 2024-04-28T12:32:25.044 INFO:teuthology.orchestra.run.smithi179.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 202 kB/s | 163 kB 00:00 2024-04-28T12:32:26.103 INFO:teuthology.orchestra.run.smithi179.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 120 kB/s | 127 kB 00:01 2024-04-28T12:32:28.345 INFO:teuthology.orchestra.run.smithi179.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 223 kB/s | 501 kB 00:02 2024-04-28T12:32:31.282 INFO:teuthology.orchestra.run.smithi161.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 267 kB/s | 4.7 MB 00:18 2024-04-28T12:32:34.137 INFO:teuthology.orchestra.run.smithi179.stdout:(18/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 365 kB/s | 17 MB 00:47 2024-04-28T12:32:34.712 INFO:teuthology.orchestra.run.smithi179.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 78 kB/s | 45 kB 00:00 2024-04-28T12:32:35.371 INFO:teuthology.orchestra.run.smithi179.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 205 kB/s | 135 kB 00:00 2024-04-28T12:32:36.255 INFO:teuthology.orchestra.run.smithi179.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 183 kB/s | 161 kB 00:00 2024-04-28T12:32:37.621 INFO:teuthology.orchestra.run.smithi179.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 235 kB/s | 321 kB 00:01 2024-04-28T12:32:39.463 INFO:teuthology.orchestra.run.smithi179.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 162 kB/s | 298 kB 00:01 2024-04-28T12:32:39.963 INFO:teuthology.orchestra.run.smithi179.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 199 kB/s | 99 kB 00:00 2024-04-28T12:32:40.305 INFO:teuthology.orchestra.run.smithi179.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 249 kB/s | 85 kB 00:00 2024-04-28T12:32:43.847 INFO:teuthology.orchestra.run.smithi179.stdout:(26/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 355 kB/s | 5.4 MB 00:15 2024-04-28T12:32:44.247 INFO:teuthology.orchestra.run.smithi179.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 427 kB/s | 171 kB 00:00 2024-04-28T12:32:44.964 INFO:teuthology.orchestra.run.smithi179.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 48 kB/s | 34 kB 00:00 2024-04-28T12:32:45.415 INFO:teuthology.orchestra.run.smithi179.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 316 kB/s | 142 kB 00:00 2024-04-28T12:32:51.839 INFO:teuthology.orchestra.run.smithi179.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 274 kB/s | 3.1 MB 00:11 2024-04-28T12:32:52.461 INFO:teuthology.orchestra.run.smithi084.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 365 kB/s | 11 MB 00:29 2024-04-28T12:32:52.562 INFO:teuthology.orchestra.run.smithi084.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 255 kB/s | 25 kB 00:00 2024-04-28T12:32:52.912 INFO:teuthology.orchestra.run.smithi084.stdout:(10/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 472 kB/s | 17 MB 00:36 2024-04-28T12:32:53.720 INFO:teuthology.orchestra.run.smithi084.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 278 kB/s | 225 kB 00:00 2024-04-28T12:32:53.912 INFO:teuthology.orchestra.run.smithi084.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 141 kB/s | 27 kB 00:00 2024-04-28T12:32:54.187 INFO:teuthology.orchestra.run.smithi084.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 116 kB/s | 32 kB 00:00 2024-04-28T12:32:57.437 INFO:teuthology.orchestra.run.smithi084.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 230 kB/s | 747 kB 00:03 2024-04-28T12:32:57.996 INFO:teuthology.orchestra.run.smithi084.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 293 kB/s | 163 kB 00:00 2024-04-28T12:32:58.264 INFO:teuthology.orchestra.run.smithi179.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 282 kB/s | 3.5 MB 00:12 2024-04-28T12:32:58.554 INFO:teuthology.orchestra.run.smithi161.stdout:(8/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 395 kB/s | 21 MB 00:54 2024-04-28T12:32:58.645 INFO:teuthology.orchestra.run.smithi084.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 196 kB/s | 127 kB 00:00 2024-04-28T12:32:58.745 INFO:teuthology.orchestra.run.smithi161.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 131 kB/s | 25 kB 00:00 2024-04-28T12:32:59.239 INFO:teuthology.orchestra.run.smithi179.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 255 kB/s | 249 kB 00:00 2024-04-28T12:32:59.448 INFO:teuthology.orchestra.run.smithi179.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 237 kB/s | 49 kB 00:00 2024-04-28T12:32:59.565 INFO:teuthology.orchestra.run.smithi179.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 141 kB/s | 16 kB 00:00 2024-04-28T12:33:00.337 INFO:teuthology.orchestra.run.smithi084.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 296 kB/s | 501 kB 00:01 2024-04-28T12:33:00.615 INFO:teuthology.orchestra.run.smithi179.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 260 kB/s | 273 kB 00:01 2024-04-28T12:33:04.231 INFO:teuthology.orchestra.run.smithi179.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 211 kB/s | 762 kB 00:03 2024-04-28T12:33:04.923 INFO:teuthology.orchestra.run.smithi179.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 26 kB/s | 18 kB 00:00 2024-04-28T12:33:05.170 INFO:teuthology.orchestra.run.smithi161.stdout:(10/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 355 kB/s | 17 MB 00:48 2024-04-28T12:33:05.624 INFO:teuthology.orchestra.run.smithi179.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 115 kB/s | 80 kB 00:00 2024-04-28T12:33:05.812 INFO:teuthology.orchestra.run.smithi161.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 350 kB/s | 225 kB 00:00 2024-04-28T12:33:05.995 INFO:teuthology.orchestra.run.smithi161.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 147 kB/s | 27 kB 00:00 2024-04-28T12:33:06.171 INFO:teuthology.orchestra.run.smithi161.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 181 kB/s | 32 kB 00:00 2024-04-28T12:33:07.266 INFO:teuthology.orchestra.run.smithi179.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 489 kB/s | 803 kB 00:01 2024-04-28T12:33:08.133 INFO:teuthology.orchestra.run.smithi179.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 225 kB/s | 195 kB 00:00 2024-04-28T12:33:08.229 INFO:teuthology.orchestra.run.smithi161.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 363 kB/s | 747 kB 00:02 2024-04-28T12:33:08.971 INFO:teuthology.orchestra.run.smithi161.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 220 kB/s | 163 kB 00:00 2024-04-28T12:33:09.324 INFO:teuthology.orchestra.run.smithi179.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 212 kB/s | 253 kB 00:01 2024-04-28T12:33:09.379 INFO:teuthology.orchestra.run.smithi161.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 311 kB/s | 127 kB 00:00 2024-04-28T12:33:10.738 INFO:teuthology.orchestra.run.smithi161.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 369 kB/s | 501 kB 00:01 2024-04-28T12:33:10.786 INFO:teuthology.orchestra.run.smithi084.stdout:(18/152): ceph-common-19.0.0-3306.gbc612ffb.el9 335 kB/s | 21 MB 01:04 2024-04-28T12:33:11.102 INFO:teuthology.orchestra.run.smithi084.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 143 kB/s | 45 kB 00:00 2024-04-28T12:33:11.700 INFO:teuthology.orchestra.run.smithi179.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 529 kB/s | 1.2 MB 00:02 2024-04-28T12:33:11.703 INFO:teuthology.orchestra.run.smithi084.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 225 kB/s | 135 kB 00:00 2024-04-28T12:33:12.129 INFO:teuthology.orchestra.run.smithi161.stdout:(18/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 265 kB/s | 11 MB 00:40 2024-04-28T12:33:12.463 INFO:teuthology.orchestra.run.smithi161.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 135 kB/s | 45 kB 00:00 2024-04-28T12:33:12.636 INFO:teuthology.orchestra.run.smithi084.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 173 kB/s | 161 kB 00:00 2024-04-28T12:33:12.650 INFO:teuthology.orchestra.run.smithi179.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 112 kB/s | 106 kB 00:00 2024-04-28T12:33:13.954 INFO:teuthology.orchestra.run.smithi161.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 91 kB/s | 135 kB 00:01 2024-04-28T12:33:14.025 INFO:teuthology.orchestra.run.smithi179.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 98 kB/s | 135 kB 00:01 2024-04-28T12:33:14.703 INFO:teuthology.orchestra.run.smithi084.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 155 kB/s | 321 kB 00:02 2024-04-28T12:33:15.000 INFO:teuthology.orchestra.run.smithi179.stdout:(45/152): ceph-mgr-diskprediction-local-19.0.0- 328 kB/s | 7.4 MB 00:23 2024-04-28T12:33:15.094 INFO:teuthology.orchestra.run.smithi179.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 117 kB/s | 125 kB 00:01 2024-04-28T12:33:15.376 INFO:teuthology.orchestra.run.smithi179.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 689 kB 00:00 2024-04-28T12:33:15.468 INFO:teuthology.orchestra.run.smithi179.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 10 MB/s | 943 kB 00:00 2024-04-28T12:33:15.518 INFO:teuthology.orchestra.run.smithi179.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-04-28T12:33:15.568 INFO:teuthology.orchestra.run.smithi179.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-04-28T12:33:15.627 INFO:teuthology.orchestra.run.smithi179.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 313 kB 00:00 2024-04-28T12:33:15.685 INFO:teuthology.orchestra.run.smithi179.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 562 kB/s | 33 kB 00:00 2024-04-28T12:33:15.763 INFO:teuthology.orchestra.run.smithi161.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 89 kB/s | 161 kB 00:01 2024-04-28T12:33:15.936 INFO:teuthology.orchestra.run.smithi179.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-04-28T12:33:15.995 INFO:teuthology.orchestra.run.smithi179.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 313 kB/s | 18 kB 00:00 2024-04-28T12:33:16.037 INFO:teuthology.orchestra.run.smithi179.stdout:(55/152): python3-urllib3-1.26.5-5.el9.noarch.r 207 kB/s | 215 kB 00:01 2024-04-28T12:33:16.062 INFO:teuthology.orchestra.run.smithi179.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 167 kB/s | 11 kB 00:00 2024-04-28T12:33:16.087 INFO:teuthology.orchestra.run.smithi179.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-28T12:33:16.154 INFO:teuthology.orchestra.run.smithi179.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-04-28T12:33:16.205 INFO:teuthology.orchestra.run.smithi179.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 318 kB/s | 45 kB 00:00 2024-04-28T12:33:16.230 INFO:teuthology.orchestra.run.smithi179.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 7.5 MB/s | 582 kB 00:00 2024-04-28T12:33:16.255 INFO:teuthology.orchestra.run.smithi179.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 505 kB/s | 25 kB 00:00 2024-04-28T12:33:16.280 INFO:teuthology.orchestra.run.smithi179.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-28T12:33:16.331 INFO:teuthology.orchestra.run.smithi179.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-28T12:33:16.356 INFO:teuthology.orchestra.run.smithi179.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-04-28T12:33:16.373 INFO:teuthology.orchestra.run.smithi179.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 842 kB/s | 35 kB 00:00 2024-04-28T12:33:16.607 INFO:teuthology.orchestra.run.smithi179.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-04-28T12:33:16.674 INFO:teuthology.orchestra.run.smithi179.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-04-28T12:33:16.733 INFO:teuthology.orchestra.run.smithi179.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-28T12:33:16.783 INFO:teuthology.orchestra.run.smithi179.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 8.2 MB/s | 911 kB 00:00 2024-04-28T12:33:16.808 INFO:teuthology.orchestra.run.smithi179.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-04-28T12:33:16.886 INFO:teuthology.orchestra.run.smithi084.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 136 kB/s | 298 kB 00:02 2024-04-28T12:33:16.900 INFO:teuthology.orchestra.run.smithi179.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-28T12:33:16.951 INFO:teuthology.orchestra.run.smithi179.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-04-28T12:33:17.051 INFO:teuthology.orchestra.run.smithi179.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-04-28T12:33:17.110 INFO:teuthology.orchestra.run.smithi179.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-28T12:33:17.160 INFO:teuthology.orchestra.run.smithi179.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-28T12:33:17.252 INFO:teuthology.orchestra.run.smithi179.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 6.0 MB 00:00 2024-04-28T12:33:17.277 INFO:teuthology.orchestra.run.smithi179.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 219 kB/s | 26 kB 00:00 2024-04-28T12:33:17.303 INFO:teuthology.orchestra.run.smithi179.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-28T12:33:17.328 INFO:teuthology.orchestra.run.smithi179.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-28T12:33:17.353 INFO:teuthology.orchestra.run.smithi179.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 691 kB/s | 35 kB 00:00 2024-04-28T12:33:17.404 INFO:teuthology.orchestra.run.smithi179.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-04-28T12:33:17.454 INFO:teuthology.orchestra.run.smithi179.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-28T12:33:17.478 INFO:teuthology.orchestra.run.smithi084.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 168 kB/s | 99 kB 00:00 2024-04-28T12:33:17.538 INFO:teuthology.orchestra.run.smithi179.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 5.1 MB 00:00 2024-04-28T12:33:17.554 INFO:teuthology.orchestra.run.smithi179.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 771 kB/s | 77 kB 00:00 2024-04-28T12:33:17.588 INFO:teuthology.orchestra.run.smithi179.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 840 kB/s | 42 kB 00:00 2024-04-28T12:33:17.614 INFO:teuthology.orchestra.run.smithi179.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-28T12:33:17.639 INFO:teuthology.orchestra.run.smithi179.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-28T12:33:17.673 INFO:teuthology.orchestra.run.smithi179.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.6 MB/s | 279 kB 00:00 2024-04-28T12:33:17.698 INFO:teuthology.orchestra.run.smithi179.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 868 kB/s | 51 kB 00:00 2024-04-28T12:33:17.723 INFO:teuthology.orchestra.run.smithi179.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-28T12:33:17.782 INFO:teuthology.orchestra.run.smithi179.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 712 kB/s | 42 kB 00:00 2024-04-28T12:33:17.824 INFO:teuthology.orchestra.run.smithi179.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-04-28T12:33:17.874 INFO:teuthology.orchestra.run.smithi179.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-28T12:33:17.916 INFO:teuthology.orchestra.run.smithi179.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 863 kB/s | 36 kB 00:00 2024-04-28T12:33:17.967 INFO:teuthology.orchestra.run.smithi179.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-28T12:33:18.003 INFO:teuthology.orchestra.run.smithi084.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 163 kB/s | 85 kB 00:00 2024-04-28T12:33:18.017 INFO:teuthology.orchestra.run.smithi179.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-28T12:33:18.059 INFO:teuthology.orchestra.run.smithi179.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-04-28T12:33:18.118 INFO:teuthology.orchestra.run.smithi179.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-28T12:33:18.285 INFO:teuthology.orchestra.run.smithi179.stdout:(99/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-04-28T12:33:18.302 INFO:teuthology.orchestra.run.smithi179.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 346 kB/s | 64 kB 00:00 2024-04-28T12:33:18.327 INFO:teuthology.orchestra.run.smithi179.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-28T12:33:18.453 INFO:teuthology.orchestra.run.smithi179.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 148 kB/s | 22 kB 00:00 2024-04-28T12:33:18.530 INFO:teuthology.orchestra.run.smithi179.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 4.2 MB/s | 863 kB 00:00 2024-04-28T12:33:18.745 INFO:teuthology.orchestra.run.smithi179.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.4 MB/s | 309 kB 00:00 2024-04-28T12:33:18.770 INFO:teuthology.orchestra.run.smithi179.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-04-28T12:33:18.795 INFO:teuthology.orchestra.run.smithi179.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 385 kB/s | 19 kB 00:00 2024-04-28T12:33:18.837 INFO:teuthology.orchestra.run.smithi179.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 624 kB/s | 26 kB 00:00 2024-04-28T12:33:18.879 INFO:teuthology.orchestra.run.smithi179.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-28T12:33:18.921 INFO:teuthology.orchestra.run.smithi179.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-28T12:33:18.980 INFO:teuthology.orchestra.run.smithi179.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-28T12:33:19.038 INFO:teuthology.orchestra.run.smithi161.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 98 kB/s | 321 kB 00:03 2024-04-28T12:33:19.164 INFO:teuthology.orchestra.run.smithi179.stdout:(111/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.5 MB/s | 839 kB 00:00 2024-04-28T12:33:19.272 INFO:teuthology.orchestra.run.smithi179.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-04-28T12:33:19.314 INFO:teuthology.orchestra.run.smithi179.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-04-28T12:33:19.357 INFO:teuthology.orchestra.run.smithi179.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 774 kB/s | 32 kB 00:00 2024-04-28T12:33:19.415 INFO:teuthology.orchestra.run.smithi179.stdout:(115/152): libarrow-9.0.0-11.el9.x86_64.rpm 6.8 MB/s | 4.4 MB 00:00 2024-04-28T12:33:19.457 INFO:teuthology.orchestra.run.smithi179.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-28T12:33:19.482 INFO:teuthology.orchestra.run.smithi179.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-28T12:33:19.541 INFO:teuthology.orchestra.run.smithi179.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-28T12:33:19.566 INFO:teuthology.orchestra.run.smithi179.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-04-28T12:33:19.592 INFO:teuthology.orchestra.run.smithi179.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 702 kB/s | 35 kB 00:00 2024-04-28T12:33:19.626 INFO:teuthology.orchestra.run.smithi179.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-04-28T12:33:19.659 INFO:teuthology.orchestra.run.smithi179.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-04-28T12:33:19.701 INFO:teuthology.orchestra.run.smithi179.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-04-28T12:33:19.743 INFO:teuthology.orchestra.run.smithi179.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 425 kB/s | 18 kB 00:00 2024-04-28T12:33:19.785 INFO:teuthology.orchestra.run.smithi179.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 554 kB/s | 23 kB 00:00 2024-04-28T12:33:19.835 INFO:teuthology.orchestra.run.smithi179.stdout:(126/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.4 MB/s | 2.0 MB 00:00 2024-04-28T12:33:19.861 INFO:teuthology.orchestra.run.smithi179.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-04-28T12:33:19.886 INFO:teuthology.orchestra.run.smithi179.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-04-28T12:33:19.928 INFO:teuthology.orchestra.run.smithi179.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-04-28T12:33:19.970 INFO:teuthology.orchestra.run.smithi179.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-04-28T12:33:20.012 INFO:teuthology.orchestra.run.smithi179.stdout:(131/152): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-04-28T12:33:20.046 INFO:teuthology.orchestra.run.smithi179.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 5.5 MB/s | 1.0 MB 00:00 2024-04-28T12:33:20.071 INFO:teuthology.orchestra.run.smithi179.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-04-28T12:33:20.096 INFO:teuthology.orchestra.run.smithi179.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-04-28T12:33:20.122 INFO:teuthology.orchestra.run.smithi179.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-04-28T12:33:20.147 INFO:teuthology.orchestra.run.smithi179.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-04-28T12:33:20.172 INFO:teuthology.orchestra.run.smithi179.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-04-28T12:33:20.197 INFO:teuthology.orchestra.run.smithi179.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-04-28T12:33:20.215 INFO:teuthology.orchestra.run.smithi179.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-04-28T12:33:20.232 INFO:teuthology.orchestra.run.smithi179.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-04-28T12:33:20.249 INFO:teuthology.orchestra.run.smithi179.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.5 MB/s | 51 kB 00:00 2024-04-28T12:33:20.291 INFO:teuthology.orchestra.run.smithi179.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-04-28T12:33:20.316 INFO:teuthology.orchestra.run.smithi179.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-28T12:33:20.358 INFO:teuthology.orchestra.run.smithi179.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 481 kB/s | 20 kB 00:00 2024-04-28T12:33:20.402 INFO:teuthology.orchestra.run.smithi179.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-28T12:33:20.434 INFO:teuthology.orchestra.run.smithi179.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-28T12:33:20.567 INFO:teuthology.orchestra.run.smithi179.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 9.6 MB/s | 1.6 MB 00:00 2024-04-28T12:33:20.911 INFO:teuthology.orchestra.run.smithi084.stdout:(26/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 267 kB/s | 5.4 MB 00:20 2024-04-28T12:33:21.319 INFO:teuthology.orchestra.run.smithi084.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 419 kB/s | 171 kB 00:00 2024-04-28T12:33:21.379 INFO:teuthology.orchestra.run.smithi161.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 127 kB/s | 298 kB 00:02 2024-04-28T12:33:21.445 INFO:teuthology.orchestra.run.smithi084.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 274 kB/s | 34 kB 00:00 2024-04-28T12:33:21.795 INFO:teuthology.orchestra.run.smithi084.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 407 kB/s | 142 kB 00:00 2024-04-28T12:33:22.254 INFO:teuthology.orchestra.run.smithi161.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 114 kB/s | 99 kB 00:00 2024-04-28T12:33:23.621 INFO:teuthology.orchestra.run.smithi161.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 62 kB/s | 85 kB 00:01 2024-04-28T12:33:28.502 INFO:teuthology.orchestra.run.smithi084.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 301 kB/s | 3.1 MB 00:10 2024-04-28T12:33:29.646 INFO:teuthology.orchestra.run.smithi084.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 462 kB/s | 3.5 MB 00:07 2024-04-28T12:33:29.721 INFO:teuthology.orchestra.run.smithi161.stdout:(26/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 290 kB/s | 5.4 MB 00:18 2024-04-28T12:33:30.361 INFO:teuthology.orchestra.run.smithi084.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 348 kB/s | 249 kB 00:00 2024-04-28T12:33:30.536 INFO:teuthology.orchestra.run.smithi084.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 282 kB/s | 49 kB 00:00 2024-04-28T12:33:30.563 INFO:teuthology.orchestra.run.smithi161.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 203 kB/s | 171 kB 00:00 2024-04-28T12:33:30.620 INFO:teuthology.orchestra.run.smithi084.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 199 kB/s | 16 kB 00:00 2024-04-28T12:33:30.713 INFO:teuthology.orchestra.run.smithi161.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 228 kB/s | 34 kB 00:00 2024-04-28T12:33:31.328 INFO:teuthology.orchestra.run.smithi084.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 386 kB/s | 273 kB 00:00 2024-04-28T12:33:31.355 INFO:teuthology.orchestra.run.smithi161.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 222 kB/s | 142 kB 00:00 2024-04-28T12:33:32.183 INFO:teuthology.orchestra.run.smithi179.stdout:(148/152): librados2-19.0.0-3306.gbc612ffb.el9. 296 kB/s | 3.4 MB 00:11 2024-04-28T12:33:33.078 INFO:teuthology.orchestra.run.smithi084.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 436 kB/s | 762 kB 00:01 2024-04-28T12:33:33.186 INFO:teuthology.orchestra.run.smithi084.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 170 kB/s | 18 kB 00:00 2024-04-28T12:33:34.045 INFO:teuthology.orchestra.run.smithi084.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 94 kB/s | 80 kB 00:00 2024-04-28T12:33:34.925 INFO:teuthology.orchestra.run.smithi179.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 504 kB/s | 1.3 MB 00:02 2024-04-28T12:33:35.895 INFO:teuthology.orchestra.run.smithi084.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 434 kB/s | 803 kB 00:01 2024-04-28T12:33:36.335 INFO:teuthology.orchestra.run.smithi179.stdout:(150/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 204 kB/s | 3.1 MB 00:15 2024-04-28T12:33:36.446 INFO:teuthology.orchestra.run.smithi161.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 247 kB/s | 3.1 MB 00:12 2024-04-28T12:33:36.870 INFO:teuthology.orchestra.run.smithi084.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 201 kB/s | 195 kB 00:00 2024-04-28T12:33:37.895 INFO:teuthology.orchestra.run.smithi084.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 247 kB/s | 253 kB 00:01 2024-04-28T12:33:39.075 INFO:teuthology.orchestra.run.smithi179.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 591 kB/s | 2.4 MB 00:04 2024-04-28T12:33:40.553 INFO:teuthology.orchestra.run.smithi084.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 473 kB/s | 1.2 MB 00:02 2024-04-28T12:33:41.378 INFO:teuthology.orchestra.run.smithi084.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 129 kB/s | 106 kB 00:00 2024-04-28T12:33:42.287 INFO:teuthology.orchestra.run.smithi084.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 149 kB/s | 135 kB 00:00 2024-04-28T12:33:42.879 INFO:teuthology.orchestra.run.smithi161.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 314 kB/s | 3.5 MB 00:11 2024-04-28T12:33:43.395 INFO:teuthology.orchestra.run.smithi084.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 113 kB/s | 125 kB 00:01 2024-04-28T12:33:43.446 INFO:teuthology.orchestra.run.smithi161.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 439 kB/s | 249 kB 00:00 2024-04-28T12:33:43.613 INFO:teuthology.orchestra.run.smithi161.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 296 kB/s | 49 kB 00:00 2024-04-28T12:33:43.713 INFO:teuthology.orchestra.run.smithi161.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 165 kB/s | 16 kB 00:00 2024-04-28T12:33:44.280 INFO:teuthology.orchestra.run.smithi161.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 482 kB/s | 273 kB 00:00 2024-04-28T12:33:44.637 INFO:teuthology.orchestra.run.smithi084.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 173 kB/s | 215 kB 00:01 2024-04-28T12:33:45.196 INFO:teuthology.orchestra.run.smithi084.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 1.2 MB/s | 689 kB 00:00 2024-04-28T12:33:45.438 INFO:teuthology.orchestra.run.smithi161.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 658 kB/s | 762 kB 00:01 2024-04-28T12:33:45.630 INFO:teuthology.orchestra.run.smithi161.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 95 kB/s | 18 kB 00:00 2024-04-28T12:33:45.663 INFO:teuthology.orchestra.run.smithi084.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 2.0 MB/s | 943 kB 00:00 2024-04-28T12:33:45.755 INFO:teuthology.orchestra.run.smithi084.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-04-28T12:33:45.822 INFO:teuthology.orchestra.run.smithi084.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 325 kB/s | 21 kB 00:00 2024-04-28T12:33:45.881 INFO:teuthology.orchestra.run.smithi161.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 322 kB/s | 80 kB 00:00 2024-04-28T12:33:46.005 INFO:teuthology.orchestra.run.smithi084.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 1.7 MB/s | 313 kB 00:00 2024-04-28T12:33:46.080 INFO:teuthology.orchestra.run.smithi084.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 441 kB/s | 33 kB 00:00 2024-04-28T12:33:46.147 INFO:teuthology.orchestra.run.smithi161.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2024-04-28T12:33:46.223 INFO:teuthology.orchestra.run.smithi161.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-04-28T12:33:46.606 INFO:teuthology.orchestra.run.smithi161.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 661 kB/s | 253 kB 00:00 2024-04-28T12:33:46.932 INFO:teuthology.orchestra.run.smithi161.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 3.8 MB/s | 1.2 MB 00:00 2024-04-28T12:33:46.998 INFO:teuthology.orchestra.run.smithi161.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-28T12:33:47.005 INFO:teuthology.orchestra.run.smithi084.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.2 MB/s | 3.0 MB 00:00 2024-04-28T12:33:47.074 INFO:teuthology.orchestra.run.smithi161.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-04-28T12:33:47.081 INFO:teuthology.orchestra.run.smithi084.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 246 kB/s | 18 kB 00:00 2024-04-28T12:33:47.157 INFO:teuthology.orchestra.run.smithi161.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-28T12:33:47.164 INFO:teuthology.orchestra.run.smithi084.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 135 kB/s | 11 kB 00:00 2024-04-28T12:33:47.240 INFO:teuthology.orchestra.run.smithi084.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-28T12:33:47.299 INFO:teuthology.orchestra.run.smithi161.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.5 MB/s | 215 kB 00:00 2024-04-28T12:33:47.323 INFO:teuthology.orchestra.run.smithi084.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 547 kB/s | 45 kB 00:00 2024-04-28T12:33:47.557 INFO:teuthology.orchestra.run.smithi084.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 662 kB 00:00 2024-04-28T12:33:47.757 INFO:teuthology.orchestra.run.smithi084.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 2.9 MB/s | 582 kB 00:00 2024-04-28T12:33:47.849 INFO:teuthology.orchestra.run.smithi084.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 279 kB/s | 25 kB 00:00 2024-04-28T12:33:47.916 INFO:teuthology.orchestra.run.smithi161.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 1.1 MB/s | 689 kB 00:00 2024-04-28T12:33:47.949 INFO:teuthology.orchestra.run.smithi084.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 478 kB/s | 47 kB 00:00 2024-04-28T12:33:48.100 INFO:teuthology.orchestra.run.smithi084.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-04-28T12:33:48.325 INFO:teuthology.orchestra.run.smithi161.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 2.3 MB/s | 943 kB 00:00 2024-04-28T12:33:48.383 INFO:teuthology.orchestra.run.smithi084.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 667 kB/s | 188 kB 00:00 2024-04-28T12:33:48.400 INFO:teuthology.orchestra.run.smithi161.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-04-28T12:33:48.467 INFO:teuthology.orchestra.run.smithi161.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 322 kB/s | 21 kB 00:00 2024-04-28T12:33:48.550 INFO:teuthology.orchestra.run.smithi084.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 211 kB/s | 35 kB 00:00 2024-04-28T12:33:48.567 INFO:teuthology.orchestra.run.smithi161.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 3.1 MB/s | 313 kB 00:00 2024-04-28T12:33:48.634 INFO:teuthology.orchestra.run.smithi161.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-04-28T12:33:49.701 INFO:teuthology.orchestra.run.smithi161.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.8 MB/s | 3.0 MB 00:01 2024-04-28T12:33:49.777 INFO:teuthology.orchestra.run.smithi161.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 242 kB/s | 18 kB 00:00 2024-04-28T12:33:49.975 INFO:teuthology.orchestra.run.smithi084.stdout:(65/152): openblas-openmp-0.3.21-2.el9.x86_64.r 3.3 MB/s | 4.7 MB 00:01 2024-04-28T12:33:50.194 INFO:teuthology.orchestra.run.smithi161.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 27 kB/s | 11 kB 00:00 2024-04-28T12:33:50.319 INFO:teuthology.orchestra.run.smithi161.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-28T12:33:50.419 INFO:teuthology.orchestra.run.smithi161.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 453 kB/s | 45 kB 00:00 2024-04-28T12:33:50.819 INFO:teuthology.orchestra.run.smithi161.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 1.6 MB/s | 662 kB 00:00 2024-04-28T12:33:51.128 INFO:teuthology.orchestra.run.smithi161.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 1.8 MB/s | 582 kB 00:00 2024-04-28T12:33:51.228 INFO:teuthology.orchestra.run.smithi161.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 255 kB/s | 25 kB 00:00 2024-04-28T12:33:51.312 INFO:teuthology.orchestra.run.smithi161.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 568 kB/s | 47 kB 00:00 2024-04-28T12:33:51.400 INFO:teuthology.orchestra.run.smithi084.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.1 MB/s | 4.4 MB 00:01 2024-04-28T12:33:51.495 INFO:teuthology.orchestra.run.smithi161.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-04-28T12:33:51.604 INFO:teuthology.orchestra.run.smithi161.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-28T12:33:51.663 INFO:teuthology.orchestra.run.smithi161.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 606 kB/s | 35 kB 00:00 2024-04-28T12:33:52.017 INFO:teuthology.orchestra.run.smithi084.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 2.6 MB/s | 1.6 MB 00:00 2024-04-28T12:33:52.151 INFO:teuthology.orchestra.run.smithi084.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 6.7 MB/s | 911 kB 00:00 2024-04-28T12:33:52.210 INFO:teuthology.orchestra.run.smithi084.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-28T12:33:52.901 INFO:teuthology.orchestra.run.smithi084.stdout:(70/152): postgresql-server-13.14-1.el9.x86_64. 8.4 MB/s | 5.8 MB 00:00 2024-04-28T12:33:53.026 INFO:teuthology.orchestra.run.smithi084.stdout:(71/152): ceph-mgr-diskprediction-local-19.0.0- 309 kB/s | 7.4 MB 00:24 2024-04-28T12:33:53.138 INFO:teuthology.orchestra.run.smithi161.stdout:(65/152): openblas-openmp-0.3.21-2.el9.x86_64.r 3.2 MB/s | 4.7 MB 00:01 2024-04-28T12:33:53.143 INFO:teuthology.orchestra.run.smithi084.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-04-28T12:33:53.286 INFO:teuthology.orchestra.run.smithi084.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 339 kB/s | 87 kB 00:00 2024-04-28T12:33:53.494 INFO:teuthology.orchestra.run.smithi084.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.1 MB/s | 244 kB 00:00 2024-04-28T12:33:53.594 INFO:teuthology.orchestra.run.smithi084.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-28T12:33:53.703 INFO:teuthology.orchestra.run.smithi084.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 238 kB/s | 26 kB 00:00 2024-04-28T12:33:53.787 INFO:teuthology.orchestra.run.smithi084.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 232 kB/s | 19 kB 00:00 2024-04-28T12:33:53.887 INFO:teuthology.orchestra.run.smithi084.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-04-28T12:33:53.979 INFO:teuthology.orchestra.run.smithi084.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 381 kB/s | 35 kB 00:00 2024-04-28T12:33:54.171 INFO:teuthology.orchestra.run.smithi084.stdout:(80/152): python3-babel-2.9.1-2.el9.noarch.rpm 5.8 MB/s | 6.0 MB 00:01 2024-04-28T12:33:54.279 INFO:teuthology.orchestra.run.smithi084.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-28T12:33:54.371 INFO:teuthology.orchestra.run.smithi084.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-04-28T12:33:54.438 INFO:teuthology.orchestra.run.smithi084.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-04-28T12:33:54.505 INFO:teuthology.orchestra.run.smithi084.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 639 kB/s | 42 kB 00:00 2024-04-28T12:33:54.580 INFO:teuthology.orchestra.run.smithi084.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-04-28T12:33:54.723 INFO:teuthology.orchestra.run.smithi084.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-04-28T12:33:54.798 INFO:teuthology.orchestra.run.smithi084.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.7 MB/s | 279 kB 00:00 2024-04-28T12:33:54.857 INFO:teuthology.orchestra.run.smithi084.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 881 kB/s | 51 kB 00:00 2024-04-28T12:33:54.906 INFO:teuthology.orchestra.run.smithi084.stdout:(89/152): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-04-28T12:33:55.254 INFO:teuthology.orchestra.run.smithi161.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 2.1 MB/s | 4.4 MB 00:02 2024-04-28T12:33:55.482 INFO:teuthology.orchestra.run.smithi084.stdout:(90/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.4 MB/s | 5.1 MB 00:01 2024-04-28T12:33:55.557 INFO:teuthology.orchestra.run.smithi084.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 560 kB/s | 42 kB 00:00 2024-04-28T12:33:55.633 INFO:teuthology.orchestra.run.smithi084.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 592 kB/s | 44 kB 00:00 2024-04-28T12:33:55.733 INFO:teuthology.orchestra.run.smithi084.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.2 MB/s | 221 kB 00:00 2024-04-28T12:33:55.825 INFO:teuthology.orchestra.run.smithi084.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 395 kB/s | 36 kB 00:00 2024-04-28T12:33:55.917 INFO:teuthology.orchestra.run.smithi084.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 522 kB/s | 47 kB 00:00 2024-04-28T12:33:56.076 INFO:teuthology.orchestra.run.smithi084.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.9 MB/s | 305 kB 00:00 2024-04-28T12:33:56.180 INFO:teuthology.orchestra.run.smithi161.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 1.7 MB/s | 1.6 MB 00:00 2024-04-28T12:33:56.284 INFO:teuthology.orchestra.run.smithi084.stdout:(97/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 12 MB/s | 16 MB 00:01 2024-04-28T12:33:56.309 INFO:teuthology.orchestra.run.smithi084.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 248 kB/s | 58 kB 00:00 2024-04-28T12:33:56.370 INFO:teuthology.orchestra.run.smithi084.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-04-28T12:33:56.396 INFO:teuthology.orchestra.run.smithi084.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 748 kB/s | 64 kB 00:00 2024-04-28T12:33:56.471 INFO:teuthology.orchestra.run.smithi084.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 449 kB/s | 45 kB 00:00 2024-04-28T12:33:56.680 INFO:teuthology.orchestra.run.smithi161.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 1.8 MB/s | 911 kB 00:00 2024-04-28T12:33:56.797 INFO:teuthology.orchestra.run.smithi161.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.1 MB/s | 133 kB 00:00 2024-04-28T12:33:57.213 INFO:teuthology.orchestra.run.smithi084.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 27 kB/s | 22 kB 00:00 2024-04-28T12:33:58.021 INFO:teuthology.orchestra.run.smithi084.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 679 kB/s | 548 kB 00:00 2024-04-28T12:33:58.230 INFO:teuthology.orchestra.run.smithi084.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.5 MB/s | 309 kB 00:00 2024-04-28T12:33:58.321 INFO:teuthology.orchestra.run.smithi084.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 213 kB/s | 19 kB 00:00 2024-04-28T12:33:58.838 INFO:teuthology.orchestra.run.smithi084.stdout:(106/152): protobuf-compiler-3.14.0-13.el9.x86_ 365 kB/s | 863 kB 00:02 2024-04-28T12:33:59.730 INFO:teuthology.orchestra.run.smithi084.stdout:(107/152): libarrow-9.0.0-11.el9.x86_64.rpm 3.1 MB/s | 4.4 MB 00:01 2024-04-28T12:33:59.755 INFO:teuthology.orchestra.run.smithi084.stdout:(108/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 28 kB/s | 26 kB 00:00 2024-04-28T12:33:59.814 INFO:teuthology.orchestra.run.smithi084.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 583 kB/s | 49 kB 00:00 2024-04-28T12:33:59.839 INFO:teuthology.orchestra.run.smithi084.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 814 kB/s | 67 kB 00:00 2024-04-28T12:33:59.906 INFO:teuthology.orchestra.run.smithi084.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-04-28T12:34:00.148 INFO:teuthology.orchestra.run.smithi084.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.7 MB/s | 839 kB 00:00 2024-04-28T12:34:00.163 INFO:teuthology.orchestra.run.smithi161.stdout:(70/152): postgresql-server-13.14-1.el9.x86_64. 1.7 MB/s | 5.8 MB 00:03 2024-04-28T12:34:00.232 INFO:teuthology.orchestra.run.smithi084.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 1.6 MB/s | 548 kB 00:00 2024-04-28T12:34:00.257 INFO:teuthology.orchestra.run.smithi084.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 416 kB/s | 43 kB 00:00 2024-04-28T12:34:00.316 INFO:teuthology.orchestra.run.smithi084.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 391 kB/s | 32 kB 00:00 2024-04-28T12:34:00.358 INFO:teuthology.orchestra.run.smithi084.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-04-28T12:34:00.399 INFO:teuthology.orchestra.run.smithi084.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-28T12:34:00.525 INFO:teuthology.orchestra.run.smithi084.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.1 MB/s | 358 kB 00:00 2024-04-28T12:34:00.625 INFO:teuthology.orchestra.run.smithi084.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.3 MB/s | 303 kB 00:00 2024-04-28T12:34:00.642 INFO:teuthology.orchestra.run.smithi084.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 306 kB/s | 35 kB 00:00 2024-04-28T12:34:00.700 INFO:teuthology.orchestra.run.smithi084.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-28T12:34:00.913 INFO:teuthology.orchestra.run.smithi161.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 1.3 MB/s | 1.0 MB 00:00 2024-04-28T12:34:01.039 INFO:teuthology.orchestra.run.smithi161.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 700 kB/s | 87 kB 00:00 2024-04-28T12:34:01.084 INFO:teuthology.orchestra.run.smithi084.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 4.6 MB/s | 2.0 MB 00:00 2024-04-28T12:34:01.109 INFO:teuthology.orchestra.run.smithi084.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 354 kB/s | 144 kB 00:00 2024-04-28T12:34:01.185 INFO:teuthology.orchestra.run.smithi084.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 109 kB/s | 11 kB 00:00 2024-04-28T12:34:01.210 INFO:teuthology.orchestra.run.smithi084.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-04-28T12:34:01.252 INFO:teuthology.orchestra.run.smithi084.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 351 kB/s | 23 kB 00:00 2024-04-28T12:34:01.286 INFO:teuthology.orchestra.run.smithi084.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 261 kB/s | 19 kB 00:00 2024-04-28T12:34:01.336 INFO:teuthology.orchestra.run.smithi084.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 236 kB/s | 20 kB 00:00 2024-04-28T12:34:01.544 INFO:teuthology.orchestra.run.smithi084.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 4.0 MB/s | 1.0 MB 00:00 2024-04-28T12:34:01.570 INFO:teuthology.orchestra.run.smithi084.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 199 kB/s | 46 kB 00:00 2024-04-28T12:34:01.604 INFO:teuthology.orchestra.run.smithi084.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-28T12:34:01.629 INFO:teuthology.orchestra.run.smithi084.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 987 kB/s | 58 kB 00:00 2024-04-28T12:34:01.671 INFO:teuthology.orchestra.run.smithi084.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-28T12:34:01.704 INFO:teuthology.orchestra.run.smithi084.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 220 kB/s | 16 kB 00:00 2024-04-28T12:34:01.755 INFO:teuthology.orchestra.run.smithi084.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-04-28T12:34:01.780 INFO:teuthology.orchestra.run.smithi084.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 377 kB/s | 28 kB 00:00 2024-04-28T12:34:01.830 INFO:teuthology.orchestra.run.smithi084.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 412 kB/s | 31 kB 00:00 2024-04-28T12:34:01.872 INFO:teuthology.orchestra.run.smithi084.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-04-28T12:34:01.923 INFO:teuthology.orchestra.run.smithi084.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 648 kB/s | 59 kB 00:00 2024-04-28T12:34:01.964 INFO:teuthology.orchestra.run.smithi084.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 392 kB/s | 36 kB 00:00 2024-04-28T12:34:02.006 INFO:teuthology.orchestra.run.smithi084.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 615 kB/s | 51 kB 00:00 2024-04-28T12:34:02.040 INFO:teuthology.orchestra.run.smithi084.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-28T12:34:02.065 INFO:teuthology.orchestra.run.smithi084.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.6 MB/s | 90 kB 00:00 2024-04-28T12:34:02.149 INFO:teuthology.orchestra.run.smithi084.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.9 MB/s | 427 kB 00:00 2024-04-28T12:34:02.165 INFO:teuthology.orchestra.run.smithi084.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 201 kB/s | 20 kB 00:00 2024-04-28T12:34:02.241 INFO:teuthology.orchestra.run.smithi084.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-28T12:34:02.516 INFO:teuthology.orchestra.run.smithi084.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 4.5 MB/s | 1.6 MB 00:00 2024-04-28T12:34:05.247 INFO:teuthology.orchestra.run.smithi161.stdout:(73/152): python3-babel-2.9.1-2.el9.noarch.rpm 1.4 MB/s | 6.0 MB 00:04 2024-04-28T12:34:05.530 INFO:teuthology.orchestra.run.smithi161.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 862 kB/s | 244 kB 00:00 2024-04-28T12:34:05.731 INFO:teuthology.orchestra.run.smithi161.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.2 MB/s | 248 kB 00:00 2024-04-28T12:34:05.806 INFO:teuthology.orchestra.run.smithi161.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 342 kB/s | 26 kB 00:00 2024-04-28T12:34:05.873 INFO:teuthology.orchestra.run.smithi161.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 287 kB/s | 19 kB 00:00 2024-04-28T12:34:06.048 INFO:teuthology.orchestra.run.smithi161.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 984 kB/s | 172 kB 00:00 2024-04-28T12:34:06.149 INFO:teuthology.orchestra.run.smithi161.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 347 kB/s | 35 kB 00:00 2024-04-28T12:34:08.315 INFO:teuthology.orchestra.run.smithi084.stdout:(148/152): librados2-19.0.0-3306.gbc612ffb.el9. 573 kB/s | 3.4 MB 00:06 2024-04-28T12:34:09.065 INFO:teuthology.orchestra.run.smithi161.stdout:(80/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.7 MB/s | 5.1 MB 00:02 2024-04-28T12:34:09.249 INFO:teuthology.orchestra.run.smithi161.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 250 kB 00:00 2024-04-28T12:34:09.382 INFO:teuthology.orchestra.run.smithi161.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-04-28T12:34:09.450 INFO:teuthology.orchestra.run.smithi161.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-04-28T12:34:09.525 INFO:teuthology.orchestra.run.smithi161.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 563 kB/s | 42 kB 00:00 2024-04-28T12:34:09.717 INFO:teuthology.orchestra.run.smithi161.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 1.4 MB/s | 269 kB 00:00 2024-04-28T12:34:09.867 INFO:teuthology.orchestra.run.smithi161.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-04-28T12:34:10.176 INFO:teuthology.orchestra.run.smithi161.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 904 kB/s | 279 kB 00:00 2024-04-28T12:34:10.226 INFO:teuthology.orchestra.run.smithi161.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-28T12:34:10.276 INFO:teuthology.orchestra.run.smithi161.stdout:(89/152): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-04-28T12:34:11.165 INFO:teuthology.orchestra.run.smithi084.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 485 kB/s | 1.3 MB 00:02 2024-04-28T12:34:13.093 INFO:teuthology.orchestra.run.smithi161.stdout:(90/152): ceph-mgr-diskprediction-local-19.0.0- 207 kB/s | 7.4 MB 00:36 2024-04-28T12:34:13.267 INFO:teuthology.orchestra.run.smithi161.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 239 kB/s | 42 kB 00:00 2024-04-28T12:34:13.318 INFO:teuthology.orchestra.run.smithi161.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 874 kB/s | 44 kB 00:00 2024-04-28T12:34:13.430 INFO:teuthology.orchestra.run.smithi161.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-04-28T12:34:13.477 INFO:teuthology.orchestra.run.smithi161.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 721 kB/s | 36 kB 00:00 2024-04-28T12:34:13.527 INFO:teuthology.orchestra.run.smithi161.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-04-28T12:34:13.636 INFO:teuthology.orchestra.run.smithi161.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 305 kB 00:00 2024-04-28T12:34:13.686 INFO:teuthology.orchestra.run.smithi161.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-28T12:34:13.753 INFO:teuthology.orchestra.run.smithi161.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-04-28T12:34:13.820 INFO:teuthology.orchestra.run.smithi161.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 955 kB/s | 64 kB 00:00 2024-04-28T12:34:13.887 INFO:teuthology.orchestra.run.smithi161.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 672 kB/s | 45 kB 00:00 2024-04-28T12:34:14.187 INFO:teuthology.orchestra.run.smithi161.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-04-28T12:34:14.746 INFO:teuthology.orchestra.run.smithi161.stdout:(102/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.5 MB/s | 863 kB 00:00 2024-04-28T12:34:15.346 INFO:teuthology.orchestra.run.smithi161.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 914 kB/s | 548 kB 00:00 2024-04-28T12:34:15.855 INFO:teuthology.orchestra.run.smithi161.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 608 kB/s | 309 kB 00:00 2024-04-28T12:34:15.906 INFO:teuthology.orchestra.run.smithi084.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 517 kB/s | 2.4 MB 00:04 2024-04-28T12:34:16.113 INFO:teuthology.orchestra.run.smithi161.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-04-28T12:34:16.872 INFO:teuthology.orchestra.run.smithi161.stdout:(106/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.5 MB/s | 16 MB 00:06 2024-04-28T12:34:17.097 INFO:teuthology.orchestra.run.smithi161.stdout:(107/152): libarrow-9.0.0-11.el9.x86_64.rpm 4.5 MB/s | 4.4 MB 00:00 2024-04-28T12:34:17.131 INFO:teuthology.orchestra.run.smithi161.stdout:(108/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 101 kB/s | 26 kB 00:00 2024-04-28T12:34:17.431 INFO:teuthology.orchestra.run.smithi161.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 146 kB/s | 49 kB 00:00 2024-04-28T12:34:17.473 INFO:teuthology.orchestra.run.smithi161.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 197 kB/s | 67 kB 00:00 2024-04-28T12:34:17.882 INFO:teuthology.orchestra.run.smithi161.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 335 kB/s | 151 kB 00:00 2024-04-28T12:34:18.157 INFO:teuthology.orchestra.run.smithi161.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.2 MB/s | 839 kB 00:00 2024-04-28T12:34:18.248 INFO:teuthology.orchestra.run.smithi084.stdout:(151/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 204 kB/s | 3.1 MB 00:15 2024-04-28T12:34:18.474 INFO:teuthology.orchestra.run.smithi161.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 926 kB/s | 548 kB 00:00 2024-04-28T12:34:18.499 INFO:teuthology.orchestra.run.smithi161.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 127 kB/s | 43 kB 00:00 2024-04-28T12:34:18.741 INFO:teuthology.orchestra.run.smithi161.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 121 kB/s | 32 kB 00:00 2024-04-28T12:34:18.766 INFO:teuthology.orchestra.run.smithi161.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 56 kB/s | 15 kB 00:00 2024-04-28T12:34:19.174 INFO:teuthology.orchestra.run.smithi161.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 396 kB/s | 172 kB 00:00 2024-04-28T12:34:19.283 INFO:teuthology.orchestra.run.smithi161.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 693 kB/s | 358 kB 00:00 2024-04-28T12:34:19.559 INFO:teuthology.orchestra.run.smithi161.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 128 kB/s | 35 kB 00:00 2024-04-28T12:34:19.684 INFO:teuthology.orchestra.run.smithi161.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 596 kB/s | 303 kB 00:00 2024-04-28T12:34:20.067 INFO:teuthology.orchestra.run.smithi161.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 404 kB/s | 206 kB 00:00 2024-04-28T12:34:20.459 INFO:teuthology.orchestra.run.smithi161.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.6 MB/s | 2.0 MB 00:00 2024-04-28T12:34:20.493 INFO:teuthology.orchestra.run.smithi161.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 340 kB/s | 144 kB 00:00 2024-04-28T12:34:20.643 INFO:teuthology.orchestra.run.smithi161.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 58 kB/s | 11 kB 00:00 2024-04-28T12:34:20.760 INFO:teuthology.orchestra.run.smithi161.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 66 kB/s | 18 kB 00:00 2024-04-28T12:34:20.918 INFO:teuthology.orchestra.run.smithi161.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 84 kB/s | 23 kB 00:00 2024-04-28T12:34:21.035 INFO:teuthology.orchestra.run.smithi161.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 71 kB/s | 19 kB 00:00 2024-04-28T12:34:21.186 INFO:teuthology.orchestra.run.smithi161.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 74 kB/s | 20 kB 00:00 2024-04-28T12:34:21.536 INFO:teuthology.orchestra.run.smithi161.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 132 kB/s | 46 kB 00:00 2024-04-28T12:34:21.728 INFO:teuthology.orchestra.run.smithi161.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 1.5 MB/s | 1.0 MB 00:00 2024-04-28T12:34:21.904 INFO:teuthology.orchestra.run.smithi161.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 215 kB/s | 79 kB 00:00 2024-04-28T12:34:22.070 INFO:teuthology.orchestra.run.smithi161.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 169 kB/s | 58 kB 00:00 2024-04-28T12:34:22.337 INFO:teuthology.orchestra.run.smithi161.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 61 kB/s | 16 kB 00:00 2024-04-28T12:34:22.412 INFO:teuthology.orchestra.run.smithi161.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 535 kB/s | 272 kB 00:00 2024-04-28T12:34:22.679 INFO:teuthology.orchestra.run.smithi161.stdout:(135/152): python3-pyperclip-1.8.2-2.el9.noarch 106 kB/s | 28 kB 00:00 2024-04-28T12:34:22.704 INFO:teuthology.orchestra.run.smithi161.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 245 kB/s | 90 kB 00:00 2024-04-28T12:34:22.946 INFO:teuthology.orchestra.run.smithi161.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 115 kB/s | 31 kB 00:00 2024-04-28T12:34:23.205 INFO:teuthology.orchestra.run.smithi161.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 377 kB/s | 188 kB 00:00 2024-04-28T12:34:23.288 INFO:teuthology.orchestra.run.smithi161.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-04-28T12:34:23.489 INFO:teuthology.orchestra.run.smithi161.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 126 kB/s | 36 kB 00:00 2024-04-28T12:34:23.631 INFO:teuthology.orchestra.run.smithi161.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 149 kB/s | 51 kB 00:00 2024-04-28T12:34:23.989 INFO:teuthology.orchestra.run.smithi161.stdout:(142/152): python3-websocket-client-1.2.3-2.el9 250 kB/s | 90 kB 00:00 2024-04-28T12:34:24.015 INFO:teuthology.orchestra.run.smithi161.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 437 kB/s | 230 kB 00:00 2024-04-28T12:34:24.281 INFO:teuthology.orchestra.run.smithi161.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 75 kB/s | 20 kB 00:00 2024-04-28T12:34:24.573 INFO:teuthology.orchestra.run.smithi161.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 732 kB/s | 427 kB 00:00 2024-04-28T12:34:24.724 INFO:teuthology.orchestra.run.smithi161.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 434 kB/s | 192 kB 00:00 2024-04-28T12:34:25.257 INFO:teuthology.orchestra.run.smithi161.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 2.3 MB/s | 1.6 MB 00:00 2024-04-28T12:34:28.624 INFO:teuthology.orchestra.run.smithi161.stdout:(148/152): librados2-19.0.0-3306.gbc612ffb.el9. 892 kB/s | 3.4 MB 00:03 2024-04-28T12:34:29.299 INFO:teuthology.orchestra.run.smithi161.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 2.0 MB/s | 1.3 MB 00:00 2024-04-28T12:34:30.391 INFO:teuthology.orchestra.run.smithi161.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.2 MB/s | 2.4 MB 00:01 2024-04-28T12:34:34.040 INFO:teuthology.orchestra.run.smithi161.stdout:(151/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 366 kB/s | 3.1 MB 00:08 2024-04-28T12:34:35.611 INFO:teuthology.orchestra.run.smithi084.stdout:(152/152): ceph-test-19.0.0-3306.gbc612ffb.el9. 488 kB/s | 49 MB 01:43 2024-04-28T12:34:35.615 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-04-28T12:34:35.615 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.5 MB/s | 222 MB 02:30 2024-04-28T12:34:38.431 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-04-28T12:34:38.683 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-04-28T12:34:38.683 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-04-28T12:34:41.391 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-04-28T12:34:41.391 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-04-28T12:34:42.040 INFO:teuthology.orchestra.run.smithi161.stdout:(152/152): ceph-test-19.0.0-3306.gbc612ffb.el9. 487 kB/s | 49 MB 01:43 2024-04-28T12:34:42.043 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-04-28T12:34:42.043 INFO:teuthology.orchestra.run.smithi161.stdout:Total 1.4 MB/s | 222 MB 02:39 2024-04-28T12:34:43.084 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-04-28T12:34:43.229 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-04-28T12:34:43.229 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-04-28T12:34:45.082 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-04-28T12:34:45.082 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-04-28T12:34:47.755 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-04-28T12:34:47.895 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-28T12:34:47.981 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-28T12:34:48.090 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-28T12:34:48.153 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-28T12:34:48.193 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-28T12:34:48.321 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-28T12:34:48.757 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-28T12:34:48.866 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:34:49.216 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:34:49.242 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:34:49.364 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:34:49.408 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T12:34:49.716 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-28T12:34:49.772 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-28T12:34:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:34:49.926 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:34:49.964 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T12:34:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T12:34:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T12:34:50.160 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T12:34:50.461 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T12:34:50.514 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T12:34:50.701 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-28T12:34:50.826 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-28T12:34:50.865 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-28T12:34:50.912 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-28T12:34:51.004 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-28T12:34:51.060 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-28T12:34:51.079 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-04-28T12:34:51.122 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-28T12:34:51.160 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-28T12:34:51.173 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-28T12:34:51.198 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-28T12:34:51.213 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-28T12:34:51.251 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-28T12:34:51.286 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-28T12:34:51.311 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-28T12:34:51.329 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-28T12:34:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-28T12:34:51.416 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-28T12:34:51.448 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-28T12:34:51.505 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-28T12:34:51.611 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-28T12:34:51.652 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-28T12:34:51.692 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-28T12:34:51.705 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-28T12:34:51.740 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-28T12:34:51.796 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:34:51.854 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-28T12:34:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:34:51.911 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-28T12:34:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:34:52.021 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:34:52.045 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-28T12:34:52.059 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T12:34:52.214 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-28T12:34:52.218 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-28T12:34:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-28T12:34:52.281 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-28T12:34:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:34:52.404 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:34:52.418 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T12:34:52.481 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T12:34:52.494 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T12:34:52.513 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T12:34:52.711 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T12:34:52.744 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T12:34:52.851 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-28T12:34:52.948 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-28T12:34:52.950 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-28T12:34:52.993 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-28T12:34:53.028 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-28T12:34:53.063 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-28T12:34:53.093 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-28T12:34:53.115 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-28T12:34:53.127 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-28T12:34:53.161 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-28T12:34:53.170 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-28T12:34:53.201 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-28T12:34:53.203 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-28T12:34:53.231 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-28T12:34:53.251 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-28T12:34:53.300 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-28T12:34:53.353 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-28T12:34:53.383 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-28T12:34:53.418 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-28T12:34:53.474 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-28T12:34:53.495 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-28T12:34:53.525 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-28T12:34:53.529 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-28T12:34:53.572 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-28T12:34:53.584 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-28T12:34:53.601 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-28T12:34:53.663 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-28T12:34:53.672 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-28T12:34:53.708 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-28T12:34:53.741 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-28T12:34:53.746 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-28T12:34:53.813 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-28T12:34:53.816 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-28T12:34:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-28T12:34:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-28T12:34:53.952 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-28T12:34:53.975 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-28T12:34:54.001 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-28T12:34:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-28T12:34:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-28T12:34:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-28T12:34:54.310 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-28T12:34:54.439 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-28T12:34:54.483 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-28T12:34:54.529 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-28T12:34:54.540 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-28T12:34:54.565 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-28T12:34:54.597 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-28T12:34:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-28T12:34:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-28T12:34:54.694 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-28T12:34:54.732 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-28T12:34:54.774 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-28T12:34:54.798 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-28T12:34:54.961 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-28T12:34:54.970 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-28T12:34:55.012 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-28T12:34:55.035 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-28T12:34:55.059 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-28T12:34:55.090 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-28T12:34:55.093 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-28T12:34:55.153 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-28T12:34:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-28T12:34:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-28T12:34:55.205 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-28T12:34:55.238 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-28T12:34:55.248 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-28T12:34:55.285 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-28T12:34:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-28T12:34:55.321 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-28T12:34:55.344 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-28T12:34:55.364 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-28T12:34:55.407 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-28T12:34:55.445 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-28T12:34:55.495 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-28T12:34:55.542 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-28T12:34:55.604 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-28T12:34:55.633 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-28T12:34:55.636 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-28T12:34:55.675 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-28T12:34:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-28T12:34:55.742 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-28T12:34:55.769 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-28T12:34:55.809 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-28T12:34:55.811 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-28T12:34:55.849 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-28T12:34:55.874 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-28T12:34:55.899 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-28T12:34:55.938 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-28T12:34:55.973 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-28T12:34:56.022 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-28T12:34:56.091 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-28T12:34:56.123 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-28T12:34:56.151 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-28T12:34:56.185 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-28T12:34:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-28T12:34:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-28T12:34:56.286 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-28T12:34:56.320 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-28T12:34:56.353 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-28T12:34:56.394 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-28T12:34:56.429 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-28T12:34:56.516 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-28T12:34:56.559 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-28T12:34:56.599 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-28T12:34:56.646 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-28T12:34:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-28T12:34:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-28T12:34:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-28T12:34:57.013 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-28T12:34:57.046 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-28T12:34:57.161 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-28T12:34:57.232 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-28T12:34:57.328 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-28T12:34:57.372 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-28T12:34:57.517 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-28T12:34:57.564 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-28T12:34:57.598 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-28T12:34:57.603 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-28T12:34:57.638 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-28T12:34:57.698 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-28T12:34:57.748 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-28T12:34:57.772 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-28T12:34:57.860 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-28T12:34:57.909 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-28T12:34:58.085 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-28T12:34:58.123 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-28T12:34:58.200 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-28T12:34:58.573 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-28T12:34:58.717 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-28T12:34:58.844 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-28T12:34:58.962 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-28T12:35:00.186 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-28T12:35:00.276 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-28T12:35:00.294 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-28T12:35:00.304 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-28T12:35:00.337 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-28T12:35:00.384 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-28T12:35:00.434 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-28T12:35:00.477 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-28T12:35:00.541 INFO:teuthology.orchestra.run.smithi179.stdout:(152/152): ceph-test-19.0.0-3306.gbc612ffb.el9. 290 kB/s | 49 MB 02:53 2024-04-28T12:35:00.546 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-04-28T12:35:00.546 INFO:teuthology.orchestra.run.smithi179.stdout:Total 1.1 MB/s | 222 MB 03:18 2024-04-28T12:35:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-28T12:35:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T12:35:00.774 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T12:35:00.796 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-28T12:35:00.840 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-28T12:35:00.854 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-28T12:35:00.905 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T12:35:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T12:35:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-28T12:35:01.116 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-28T12:35:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-28T12:35:01.353 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T12:35:01.408 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T12:35:01.430 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-28T12:35:01.586 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-04-28T12:35:01.688 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-28T12:35:01.728 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-04-28T12:35:01.729 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-04-28T12:35:01.741 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T12:35:01.883 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T12:35:01.909 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-28T12:35:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-04-28T12:35:03.543 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-04-28T12:35:03.751 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:35:03.793 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:35:04.334 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:35:04.388 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T12:35:04.472 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T12:35:04.472 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-28T12:35:04.472 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-28T12:35:04.472 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:04.497 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T12:35:04.687 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:35:04.723 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:35:05.368 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:35:05.418 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T12:35:05.578 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T12:35:05.578 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-28T12:35:05.578 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-28T12:35:05.578 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:05.627 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T12:35:09.472 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-04-28T12:35:09.595 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-28T12:35:09.642 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-28T12:35:09.704 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-28T12:35:09.746 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-28T12:35:09.772 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-28T12:35:09.815 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-28T12:35:10.143 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-28T12:35:10.257 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:35:10.378 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:35:10.408 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:35:10.500 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:35:10.530 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T12:35:10.683 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-28T12:35:10.735 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-28T12:35:10.763 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:35:10.867 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:35:10.889 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T12:35:10.952 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-28T12:35:10.974 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T12:35:10.995 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-28T12:35:11.140 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T12:35:11.174 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-28T12:35:11.280 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-28T12:35:11.367 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-28T12:35:11.407 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-28T12:35:11.441 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-28T12:35:11.506 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-28T12:35:11.550 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-28T12:35:11.593 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-28T12:35:11.626 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-28T12:35:11.654 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-28T12:35:11.731 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-28T12:35:11.793 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-28T12:35:11.823 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-28T12:35:11.858 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-28T12:35:11.935 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-28T12:35:11.970 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-28T12:35:12.040 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-28T12:35:12.067 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-28T12:35:12.137 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-28T12:35:12.190 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-28T12:35:12.264 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-28T12:35:12.399 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-28T12:35:12.460 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-28T12:35:12.989 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-28T12:35:13.081 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-28T12:35:13.110 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-28T12:35:13.144 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-28T12:35:13.183 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-28T12:35:13.224 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-28T12:35:13.382 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-28T12:35:13.453 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-28T12:35:13.508 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-28T12:35:13.581 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-28T12:35:13.612 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-28T12:35:13.649 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-28T12:35:13.696 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-28T12:35:13.732 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-28T12:35:13.775 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-28T12:35:13.818 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-28T12:35:13.953 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-28T12:35:14.007 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-28T12:35:14.037 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-28T12:35:14.078 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-28T12:35:14.173 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-28T12:35:14.212 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-28T12:35:14.252 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-28T12:35:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-28T12:35:14.303 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-28T12:35:14.379 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-28T12:35:14.513 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-28T12:35:14.553 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-28T12:35:14.581 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-28T12:35:14.616 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-28T12:35:14.655 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-28T12:35:14.687 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-28T12:35:14.717 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-28T12:35:14.751 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-28T12:35:14.784 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-28T12:35:14.826 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-28T12:35:14.861 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-28T12:35:14.948 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-28T12:35:14.991 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-28T12:35:15.022 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-28T12:35:15.062 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-28T12:35:15.159 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-28T12:35:15.208 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-28T12:35:15.932 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-28T12:35:16.000 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-28T12:35:16.031 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-28T12:35:16.041 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T12:35:16.041 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /sys 2024-04-28T12:35:16.041 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /proc 2024-04-28T12:35:16.041 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /mnt 2024-04-28T12:35:16.041 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /var/tmp 2024-04-28T12:35:16.041 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /home 2024-04-28T12:35:16.041 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /root 2024-04-28T12:35:16.041 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /tmp 2024-04-28T12:35:16.042 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:16.132 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-28T12:35:16.181 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-28T12:35:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T12:35:16.215 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T12:35:16.215 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:16.215 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-28T12:35:16.215 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:16.268 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-28T12:35:16.301 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-28T12:35:16.479 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-28T12:35:16.507 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-28T12:35:16.584 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-28T12:35:17.229 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-28T12:35:17.340 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-28T12:35:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T12:35:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T12:35:17.720 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:17.720 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-28T12:35:17.720 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T12:35:17.720 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T12:35:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:17.946 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T12:35:17.968 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T12:35:17.968 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:17.969 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-28T12:35:17.969 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T12:35:17.969 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T12:35:17.969 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:18.423 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T12:35:18.446 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T12:35:18.446 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:18.446 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-28T12:35:18.446 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T12:35:18.446 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T12:35:18.446 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:18.545 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-28T12:35:18.565 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-28T12:35:18.575 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T12:35:18.575 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /sys 2024-04-28T12:35:18.575 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /proc 2024-04-28T12:35:18.576 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /mnt 2024-04-28T12:35:18.576 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /var/tmp 2024-04-28T12:35:18.576 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /home 2024-04-28T12:35:18.576 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /root 2024-04-28T12:35:18.576 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /tmp 2024-04-28T12:35:18.576 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:18.593 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-28T12:35:18.649 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-28T12:35:18.806 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T12:35:18.834 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-28T12:35:18.849 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T12:35:18.849 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:18.849 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-28T12:35:18.849 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:18.891 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-28T12:35:18.915 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-28T12:35:18.943 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-28T12:35:18.969 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-28T12:35:18.976 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-28T12:35:19.067 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T12:35:19.223 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T12:35:19.266 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-28T12:35:19.286 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-28T12:35:19.298 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-28T12:35:19.322 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T12:35:19.325 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-28T12:35:19.412 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-28T12:35:19.444 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-28T12:35:19.480 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-28T12:35:19.744 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T12:35:19.782 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T12:35:19.815 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-28T12:35:19.884 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-28T12:35:19.914 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T12:35:19.956 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-28T12:35:19.993 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T12:35:20.072 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-28T12:35:20.095 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-28T12:35:20.433 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T12:35:20.511 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T12:35:20.512 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:20.512 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-28T12:35:20.512 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T12:35:20.512 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T12:35:20.512 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:20.775 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T12:35:20.844 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T12:35:20.844 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:20.844 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-28T12:35:20.844 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T12:35:20.844 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T12:35:20.844 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T12:35:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T12:35:21.326 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T12:35:21.391 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T12:35:21.391 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:21.391 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-28T12:35:21.391 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T12:35:21.391 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T12:35:21.392 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:21.542 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-28T12:35:21.607 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-28T12:35:21.680 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-28T12:35:21.923 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-28T12:35:21.962 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T12:35:21.980 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-28T12:35:21.991 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T12:35:22.065 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-28T12:35:22.095 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T12:35:22.174 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T12:35:22.252 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-28T12:35:22.284 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T12:35:22.307 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T12:35:22.307 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:22.307 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-28T12:35:22.307 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T12:35:22.307 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T12:35:22.307 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:22.338 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T12:35:22.340 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:35:22.361 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T12:35:22.372 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T12:35:22.383 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:35:22.403 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-28T12:35:22.447 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-28T12:35:22.469 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-28T12:35:22.478 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-28T12:35:22.534 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-28T12:35:22.919 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-28T12:35:22.963 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T12:35:23.055 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-28T12:35:23.055 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-28T12:35:23.055 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-28T12:35:23.055 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:23.089 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T12:35:23.098 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T12:35:23.157 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T12:35:23.215 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-28T12:35:23.341 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-28T12:35:23.384 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T12:35:23.462 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-28T12:35:23.488 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T12:35:23.510 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T12:35:23.510 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:23.510 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-28T12:35:23.510 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T12:35:23.510 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T12:35:23.511 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:23.555 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T12:35:23.577 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T12:35:23.577 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:23.577 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-28T12:35:23.577 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:23.622 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T12:35:23.645 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T12:35:23.645 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:23.645 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-28T12:35:23.645 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T12:35:23.645 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T12:35:23.645 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:23.948 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T12:35:23.970 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T12:35:23.970 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:23.970 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-28T12:35:23.970 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T12:35:23.971 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T12:35:23.971 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:24.700 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T12:35:24.744 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T12:35:25.840 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T12:35:25.860 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T12:35:26.006 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T12:35:26.191 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-28T12:35:26.229 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T12:35:26.323 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T12:35:26.323 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:26.323 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-28T12:35:26.323 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T12:35:26.323 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T12:35:26.323 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:26.379 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T12:35:26.417 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T12:35:26.560 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-28T12:35:26.637 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-28T12:35:27.668 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-28T12:35:27.699 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T12:35:27.788 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T12:35:27.788 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:27.788 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-28T12:35:27.789 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T12:35:27.789 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T12:35:27.789 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:27.849 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T12:35:27.944 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T12:35:27.944 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:27.944 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-28T12:35:27.945 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:28.042 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T12:35:28.147 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T12:35:28.148 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:28.148 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-28T12:35:28.148 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T12:35:28.148 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T12:35:28.148 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T12:35:28.585 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-28T12:35:28.627 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-28T12:35:28.657 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T12:35:28.657 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:28.657 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-28T12:35:28.657 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T12:35:28.657 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T12:35:28.657 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:28.668 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-28T12:35:28.700 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-28T12:35:28.727 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-28T12:35:28.919 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-28T12:35:28.965 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-28T12:35:28.994 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-28T12:35:29.619 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:35:29.659 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:35:29.678 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:35:30.512 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-28T12:35:30.581 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-28T12:35:30.621 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-28T12:35:30.705 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-28T12:35:30.734 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-28T12:35:30.734 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T12:35:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T12:35:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T12:35:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T12:35:30.958 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T12:35:31.025 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:35:33.703 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-28T12:35:33.761 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-28T12:35:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-28T12:35:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-28T12:35:33.937 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-28T12:35:34.275 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-28T12:35:34.330 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-28T12:35:34.358 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-28T12:35:34.657 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-28T12:35:34.658 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /sys 2024-04-28T12:35:34.658 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /proc 2024-04-28T12:35:34.658 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /mnt 2024-04-28T12:35:34.658 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /var/tmp 2024-04-28T12:35:34.658 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /home 2024-04-28T12:35:34.658 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /root 2024-04-28T12:35:34.658 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /tmp 2024-04-28T12:35:34.658 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:34.834 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T12:35:34.848 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-28T12:35:34.848 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:34.848 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-28T12:35:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:35.272 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:35:35.347 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:35:35.403 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:35:36.296 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T12:35:36.321 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-28T12:35:36.321 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:36.321 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-28T12:35:36.321 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T12:35:36.322 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-28T12:35:36.322 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:36.393 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:35:36.393 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-28T12:35:36.393 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-28T12:35:36.393 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-28T12:35:36.393 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-28T12:35:36.393 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-28T12:35:36.393 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-28T12:35:36.394 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-28T12:35:36.394 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:35:36.394 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:35:36.394 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T12:35:36.394 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-28T12:35:36.394 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-28T12:35:36.394 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:35:36.394 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-28T12:35:36.394 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-28T12:35:36.394 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-28T12:35:36.394 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-28T12:35:36.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-28T12:35:36.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-28T12:35:36.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-28T12:35:36.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-28T12:35:36.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-28T12:35:36.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-28T12:35:36.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-28T12:35:36.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-28T12:35:36.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-28T12:35:36.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-28T12:35:36.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-28T12:35:36.395 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-28T12:35:36.407 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-28T12:35:36.407 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-28T12:35:36.408 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-28T12:35:36.409 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-28T12:35:36.410 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-28T12:35:36.411 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-28T12:35:36.411 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-28T12:35:36.411 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-28T12:35:36.411 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-28T12:35:36.411 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-28T12:35:36.411 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-28T12:35:36.411 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-28T12:35:36.411 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-28T12:35:36.411 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-28T12:35:36.412 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-28T12:35:36.412 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-28T12:35:36.412 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-28T12:35:36.412 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-28T12:35:36.412 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-28T12:35:36.412 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-28T12:35:36.412 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-28T12:35:36.412 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-28T12:35:36.412 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-28T12:35:36.412 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-28T12:35:36.412 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-28T12:35:36.412 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-28T12:35:36.413 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-28T12:35:36.414 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-28T12:35:36.415 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-28T12:35:36.416 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T12:35:36.417 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-28T12:35:36.547 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T12:35:36.569 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-28T12:35:36.569 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:36.569 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-28T12:35:36.569 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T12:35:36.569 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-28T12:35:36.569 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:36.712 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-28T12:35:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-28T12:35:36.902 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-28T12:35:37.006 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T12:35:37.027 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-28T12:35:37.028 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:37.028 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-28T12:35:37.028 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T12:35:37.028 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-28T12:35:37.028 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:37.084 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-28T12:35:37.134 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-28T12:35:37.134 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T12:35:37.136 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-28T12:35:37.184 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-28T12:35:37.249 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-28T12:35:37.261 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T12:35:37.309 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T12:35:37.309 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T12:35:37.442 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-28T12:35:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T12:35:37.491 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-28T12:35:37.560 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-28T12:35:37.580 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:35:37.650 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T12:35:37.797 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T12:35:37.833 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-28T12:35:37.865 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-28T12:35:37.873 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:35:37.873 INFO:teuthology.orchestra.run.smithi084.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-28T12:35:37.873 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:37.893 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-28T12:35:38.313 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T12:35:38.342 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-28T12:35:38.375 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-28T12:35:38.436 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-28T12:35:38.466 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T12:35:38.530 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:35:38.531 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:38.531 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2024-04-28T12:35:38.531 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.531 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.531 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-28T12:35:38.531 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-28T12:35:38.531 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-04-28T12:35:38.531 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-28T12:35:38.531 INFO:teuthology.orchestra.run.smithi161.stdout: automake-1.16.2-8.el9.noarch 2024-04-28T12:35:38.531 INFO:teuthology.orchestra.run.smithi161.stdout: bison-3.7.4-5.el9.x86_64 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:38.532 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-28T12:35:38.533 INFO:teuthology.orchestra.run.smithi161.stdout: flex-2.6.4-9.el9.x86_64 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.534 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-28T12:35:38.535 INFO:teuthology.orchestra.run.smithi161.stdout: lua-5.4.4-4.el9.x86_64 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-13.14-1.el9.x86_64 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-28T12:35:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-28T12:35:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-28T12:35:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply-3.11-14.el9.noarch 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-28T12:35:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-28T12:35:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: re2-1:20211101-3.el9.x86_64 2024-04-28T12:35:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-28T12:35:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-28T12:35:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-28T12:35:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-28T12:35:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-28T12:35:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-28T12:35:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-28T12:35:38.542 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-04-28T12:35:38.947 DEBUG:teuthology.parallel:result is None 2024-04-28T12:35:39.504 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-28T12:35:39.543 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T12:35:40.505 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-28T12:35:40.535 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T12:35:40.637 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-28T12:35:40.797 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-28T12:35:40.830 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T12:35:40.852 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-28T12:35:40.852 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:40.852 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-28T12:35:40.852 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T12:35:40.852 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-28T12:35:40.852 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:40.897 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T12:35:40.907 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-28T12:35:41.005 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-28T12:35:41.054 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-28T12:35:41.980 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-28T12:35:42.008 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T12:35:42.031 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-28T12:35:42.031 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:42.031 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-28T12:35:42.031 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T12:35:42.031 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-28T12:35:42.031 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:42.076 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T12:35:42.097 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-28T12:35:42.097 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:42.098 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-28T12:35:42.098 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:42.143 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T12:35:42.165 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-28T12:35:42.165 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:42.165 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-28T12:35:42.165 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T12:35:42.165 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-28T12:35:42.166 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:42.469 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T12:35:42.491 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-28T12:35:42.491 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-28T12:35:42.491 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-28T12:35:42.491 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T12:35:42.491 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-28T12:35:42.491 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:43.829 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-28T12:35:43.829 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-28T12:35:43.829 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-28T12:35:43.830 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-28T12:35:43.831 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-28T12:35:43.831 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-28T12:35:43.831 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-28T12:35:43.831 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-28T12:35:43.831 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-28T12:35:43.831 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-28T12:35:43.831 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-28T12:35:43.831 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-28T12:35:43.831 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-28T12:35:43.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-28T12:35:43.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-28T12:35:43.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-28T12:35:43.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-28T12:35:43.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-28T12:35:43.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-28T12:35:43.845 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-28T12:35:43.845 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-28T12:35:43.845 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-28T12:35:43.845 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-28T12:35:43.845 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-28T12:35:43.845 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-28T12:35:43.845 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-28T12:35:43.845 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-28T12:35:43.845 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-28T12:35:43.845 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-28T12:35:43.845 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-28T12:35:43.845 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-28T12:35:43.846 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-28T12:35:43.847 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-28T12:35:43.848 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-28T12:35:43.849 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-28T12:35:43.850 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-28T12:35:43.851 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-28T12:35:43.852 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-28T12:35:43.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-28T12:35:43.854 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-28T12:35:43.854 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T12:35:43.854 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-28T12:35:46.400 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:35:46.400 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:46.400 INFO:teuthology.orchestra.run.smithi084.stdout:Upgraded: 2024-04-28T12:35:46.400 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.400 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.400 INFO:teuthology.orchestra.run.smithi084.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-28T12:35:46.400 INFO:teuthology.orchestra.run.smithi084.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-28T12:35:46.400 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-04-28T12:35:46.400 INFO:teuthology.orchestra.run.smithi084.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-28T12:35:46.400 INFO:teuthology.orchestra.run.smithi084.stdout: automake-1.16.2-8.el9.noarch 2024-04-28T12:35:46.401 INFO:teuthology.orchestra.run.smithi084.stdout: bison-3.7.4-5.el9.x86_64 2024-04-28T12:35:46.401 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-28T12:35:46.401 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.401 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.401 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.401 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.401 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:46.401 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.401 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.401 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.401 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:46.402 INFO:teuthology.orchestra.run.smithi084.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.403 INFO:teuthology.orchestra.run.smithi084.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.403 INFO:teuthology.orchestra.run.smithi084.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:46.403 INFO:teuthology.orchestra.run.smithi084.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-28T12:35:46.403 INFO:teuthology.orchestra.run.smithi084.stdout: flex-2.6.4-9.el9.x86_64 2024-04-28T12:35:46.403 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-28T12:35:46.403 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-28T12:35:46.403 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-28T12:35:46.403 INFO:teuthology.orchestra.run.smithi084.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-28T12:35:46.403 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-28T12:35:46.403 INFO:teuthology.orchestra.run.smithi084.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-28T12:35:46.403 INFO:teuthology.orchestra.run.smithi084.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-28T12:35:46.403 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-28T12:35:46.404 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-28T12:35:46.404 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.404 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.404 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.404 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-28T12:35:46.404 INFO:teuthology.orchestra.run.smithi084.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-28T12:35:46.404 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-28T12:35:46.404 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-28T12:35:46.404 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: lua-5.4.4-4.el9.x86_64 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-28T12:35:46.405 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: postgresql-13.14-1.el9.x86_64 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-28T12:35:46.406 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-28T12:35:46.407 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-28T12:35:46.407 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.407 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.407 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.407 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-28T12:35:46.407 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-28T12:35:46.407 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-28T12:35:46.407 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-28T12:35:46.407 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-28T12:35:46.407 INFO:teuthology.orchestra.run.smithi084.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-28T12:35:46.407 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-28T12:35:46.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply-3.11-14.el9.noarch 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-28T12:35:46.409 INFO:teuthology.orchestra.run.smithi084.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-28T12:35:46.410 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-28T12:35:46.410 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-28T12:35:46.410 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-28T12:35:46.410 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-28T12:35:46.410 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-28T12:35:46.410 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-28T12:35:46.410 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.410 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.410 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-28T12:35:46.410 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-28T12:35:46.410 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-28T12:35:46.410 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-28T12:35:46.411 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-28T12:35:46.412 INFO:teuthology.orchestra.run.smithi084.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-28T12:35:46.412 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.412 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.412 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:46.412 INFO:teuthology.orchestra.run.smithi084.stdout: re2-1:20211101-3.el9.x86_64 2024-04-28T12:35:46.412 INFO:teuthology.orchestra.run.smithi084.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-28T12:35:46.412 INFO:teuthology.orchestra.run.smithi084.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-28T12:35:46.412 INFO:teuthology.orchestra.run.smithi084.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-28T12:35:46.412 INFO:teuthology.orchestra.run.smithi084.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-28T12:35:46.412 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-28T12:35:46.413 INFO:teuthology.orchestra.run.smithi084.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-28T12:35:46.413 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:35:46.413 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-04-28T12:35:46.935 DEBUG:teuthology.parallel:result is None 2024-04-28T12:35:47.057 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-28T12:35:47.098 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-28T12:35:47.139 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-28T12:35:47.180 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-28T12:35:47.207 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-28T12:35:47.397 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-28T12:35:47.454 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-28T12:35:47.484 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-28T12:35:48.112 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:35:48.150 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:35:48.170 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-28T12:35:48.994 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-28T12:35:49.080 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-28T12:35:49.121 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-28T12:35:49.204 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-28T12:35:49.234 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-28T12:35:49.235 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T12:35:49.302 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-28T12:35:49.335 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T12:35:49.335 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T12:35:49.416 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-28T12:35:49.483 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:35:54.986 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:35:54.986 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-28T12:35:54.986 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-28T12:35:54.986 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-28T12:35:54.986 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-28T12:35:54.986 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-28T12:35:54.986 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-28T12:35:54.987 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-28T12:35:54.988 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-28T12:35:54.988 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-28T12:35:54.988 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-28T12:35:54.988 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-28T12:35:54.988 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-28T12:35:54.988 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-28T12:35:54.988 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-28T12:35:54.988 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-28T12:35:55.000 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-28T12:35:55.000 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-28T12:35:55.000 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-28T12:35:55.000 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-28T12:35:55.000 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-28T12:35:55.000 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-28T12:35:55.000 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-28T12:35:55.000 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-28T12:35:55.000 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-28T12:35:55.000 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-28T12:35:55.000 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-28T12:35:55.000 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-28T12:35:55.001 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-28T12:35:55.002 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-28T12:35:55.003 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-28T12:35:55.004 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-28T12:35:55.005 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-28T12:35:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-28T12:35:55.007 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-28T12:35:55.008 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-28T12:35:55.009 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-28T12:35:57.144 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-28T12:35:57.144 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:57.144 INFO:teuthology.orchestra.run.smithi179.stdout:Upgraded: 2024-04-28T12:35:57.144 INFO:teuthology.orchestra.run.smithi179.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.144 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.144 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-28T12:35:57.144 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-28T12:35:57.144 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-04-28T12:35:57.144 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-28T12:35:57.144 INFO:teuthology.orchestra.run.smithi179.stdout: automake-1.16.2-8.el9.noarch 2024-04-28T12:35:57.144 INFO:teuthology.orchestra.run.smithi179.stdout: bison-3.7.4-5.el9.x86_64 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:57.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:57.146 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.146 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.146 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:57.146 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.146 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.146 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.146 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:57.146 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:57.146 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.146 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.146 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-28T12:35:57.146 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: flex-2.6.4-9.el9.x86_64 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-28T12:35:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: lua-5.4.4-4.el9.x86_64 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-13.14-1.el9.x86_64 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-28T12:35:57.149 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-28T12:35:57.150 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-28T12:35:57.151 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply-3.11-14.el9.noarch 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-28T12:35:57.152 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-28T12:35:57.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.154 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.155 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-28T12:35:57.155 INFO:teuthology.orchestra.run.smithi179.stdout: re2-1:20211101-3.el9.x86_64 2024-04-28T12:35:57.155 INFO:teuthology.orchestra.run.smithi179.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-28T12:35:57.155 INFO:teuthology.orchestra.run.smithi179.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-28T12:35:57.155 INFO:teuthology.orchestra.run.smithi179.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-28T12:35:57.155 INFO:teuthology.orchestra.run.smithi179.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-28T12:35:57.155 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-28T12:35:57.155 INFO:teuthology.orchestra.run.smithi179.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-28T12:35:57.155 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:35:57.155 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-04-28T12:35:57.582 DEBUG:teuthology.parallel:result is None 2024-04-28T12:35:57.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:35:57.809 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-28T12:35:57.862 INFO:teuthology.orchestra.run.smithi084.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-28T12:35:57.863 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-28T12:35:57.863 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-28T12:35:57.864 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:35:58.011 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-28T12:35:58.031 INFO:teuthology.orchestra.run.smithi161.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-28T12:35:58.032 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-28T12:35:58.032 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-28T12:35:58.034 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:35:58.251 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-28T12:35:58.273 INFO:teuthology.orchestra.run.smithi179.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-28T12:35:58.273 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-28T12:35:58.273 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-28T12:35:58.275 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-28T12:35:58.275 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:35:58.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-28T12:35:58.350 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-28T12:35:58.350 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-28T12:35:58.375 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-28T12:35:58.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-28T12:35:58.400 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-28T12:35:58.400 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:35:58.401 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/daemon-helper 2024-04-28T12:35:58.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-28T12:35:58.544 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-28T12:35:58.544 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2024-04-28T12:35:58.568 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-28T12:35:58.632 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-28T12:35:58.632 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/daemon-helper 2024-04-28T12:35:58.656 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-28T12:35:58.720 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-28T12:35:58.720 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:35:58.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-28T12:35:58.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-28T12:35:58.918 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-28T12:35:58.918 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-28T12:35:58.943 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-28T12:35:59.006 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-28T12:35:59.006 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-28T12:35:59.034 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-28T12:35:59.097 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-28T12:35:59.098 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:35:59.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/stdin-killer 2024-04-28T12:35:59.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-28T12:35:59.252 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-28T12:35:59.252 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/stdin-killer 2024-04-28T12:35:59.275 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-28T12:35:59.339 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-28T12:35:59.339 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/stdin-killer 2024-04-28T12:35:59.362 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-28T12:35:59.426 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-28T12:35:59.531 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-28T12:35:59.532 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:35:59.532 INFO:tasks.cephadm:Cluster fsid is ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:35:59.532 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-28T12:35:59.532 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.84', 'mon.b': '172.21.15.161', 'mon.c': '172.21.15.179'} 2024-04-28T12:35:59.532 INFO:tasks.cephadm:First mon is mon.a on smithi084 2024-04-28T12:35:59.533 INFO:tasks.cephadm:First mgr is x 2024-04-28T12:35:59.533 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-28T12:35:59.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo hostname $(hostname -s) 2024-04-28T12:35:59.625 DEBUG:teuthology.orchestra.run.smithi161:> sudo hostname $(hostname -s) 2024-04-28T12:35:59.649 DEBUG:teuthology.orchestra.run.smithi179:> sudo hostname $(hostname -s) 2024-04-28T12:35:59.672 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-28T12:35:59.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:35:59.816 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78918/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3306-gbc612ffb', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3306.gbc612ffb'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 13:02:42.256491', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240426.111959', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-28T12:36:00.035 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-testing-20240426.111959, sha1 bc612ffb9c1d5bc27536bc917a705262a36d4387 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:36:00.036 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm 2024-04-28T12:36:00.037 DEBUG:teuthology.orchestra.run.smithi084:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-28T12:36:01.750 INFO:teuthology.orchestra.run.smithi084.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 28 12:36 /home/ubuntu/cephtest/cephadm 2024-04-28T12:36:01.751 DEBUG:teuthology.orchestra.run.smithi161:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-28T12:36:03.485 INFO:teuthology.orchestra.run.smithi161.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 28 12:36 /home/ubuntu/cephtest/cephadm 2024-04-28T12:36:03.485 DEBUG:teuthology.orchestra.run.smithi179:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-28T12:36:05.198 INFO:teuthology.orchestra.run.smithi179.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 28 12:36 /home/ubuntu/cephtest/cephadm 2024-04-28T12:36:05.199 DEBUG:teuthology.orchestra.run.smithi084:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-28T12:36:05.259 DEBUG:teuthology.orchestra.run.smithi161:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-28T12:36:05.278 DEBUG:teuthology.orchestra.run.smithi179:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-28T12:36:05.309 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 on all hosts... 2024-04-28T12:36:05.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-28T12:36:05.317 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-28T12:36:05.320 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-28T12:36:06.930 INFO:teuthology.orchestra.run.smithi161.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T12:36:07.156 INFO:teuthology.orchestra.run.smithi179.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T12:36:07.205 INFO:teuthology.orchestra.run.smithi084.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T12:38:31.053 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-04-28T12:38:31.053 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-28T12:38:31.053 INFO:teuthology.orchestra.run.smithi161.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-28T12:38:31.053 INFO:teuthology.orchestra.run.smithi161.stdout: "repo_digests": [ 2024-04-28T12:38:31.054 INFO:teuthology.orchestra.run.smithi161.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-28T12:38:31.054 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-04-28T12:38:31.054 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-04-28T12:38:39.578 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-28T12:38:39.578 INFO:teuthology.orchestra.run.smithi084.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-28T12:38:39.578 INFO:teuthology.orchestra.run.smithi084.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-28T12:38:39.579 INFO:teuthology.orchestra.run.smithi084.stdout: "repo_digests": [ 2024-04-28T12:38:39.579 INFO:teuthology.orchestra.run.smithi084.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-28T12:38:39.579 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-04-28T12:38:39.579 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-28T12:39:09.296 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-04-28T12:39:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-28T12:39:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-28T12:39:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "repo_digests": [ 2024-04-28T12:39:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-28T12:39:09.297 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-04-28T12:39:09.297 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-04-28T12:39:09.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph 2024-04-28T12:39:09.417 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph 2024-04-28T12:39:09.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph 2024-04-28T12:39:09.475 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /etc/ceph 2024-04-28T12:39:09.561 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /etc/ceph 2024-04-28T12:39:09.588 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /etc/ceph 2024-04-28T12:39:09.613 INFO:tasks.cephadm:Writing seed config... 2024-04-28T12:39:09.614 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-28T12:39:09.614 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-28T12:39:09.614 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-28T12:39:09.614 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-28T12:39:09.614 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-28T12:39:09.614 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-28T12:39:09.614 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-28T12:39:09.614 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-28T12:39:09.614 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-28T12:39:09.614 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-28T12:39:09.615 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-28T12:39:09.616 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-28T12:39:09.617 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-28T12:39:09.617 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-28T12:39:09.617 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-04-28T12:39:09.617 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-28T12:39:09.617 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-28T12:39:09.617 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-28T12:39:09.617 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-28T12:39:09.618 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:39:09.618 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-28T12:39:09.673 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ddee604a-055b-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-28T12:39:09.674 DEBUG:teuthology.orchestra.run.smithi084:mon.a> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.a.service 2024-04-28T12:39:09.721 DEBUG:teuthology.orchestra.run.smithi084:mgr.x> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.x.service 2024-04-28T12:39:09.767 INFO:tasks.cephadm:Bootstrapping... 2024-04-28T12:39:09.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 -v bootstrap --fsid ddee604a-055b-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.84 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-28T12:39:10.030 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-04-28T12:39:10.031 INFO:teuthology.orchestra.run.smithi084.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387', '-v', 'bootstrap', '--fsid', 'ddee604a-055b-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.84', '--skip-admin-label'] 2024-04-28T12:39:10.031 INFO:teuthology.orchestra.run.smithi084.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-28T12:39:10.031 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying podman|docker is present... 2024-04-28T12:39:10.088 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-28T12:39:10.088 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying lvm2 is present... 2024-04-28T12:39:10.088 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying time synchronization is in place... 2024-04-28T12:39:10.105 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-28T12:39:10.105 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-28T12:39:10.121 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-28T12:39:10.121 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-04-28T12:39:10.139 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout enabled 2024-04-28T12:39:10.160 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout active 2024-04-28T12:39:10.160 INFO:teuthology.orchestra.run.smithi084.stdout:Unit chronyd.service is enabled and running 2024-04-28T12:39:10.160 INFO:teuthology.orchestra.run.smithi084.stdout:Repeating the final host check... 2024-04-28T12:39:10.221 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-28T12:39:10.221 INFO:teuthology.orchestra.run.smithi084.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-28T12:39:10.221 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl is present 2024-04-28T12:39:10.221 INFO:teuthology.orchestra.run.smithi084.stdout:lvcreate is present 2024-04-28T12:39:10.236 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-28T12:39:10.237 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-28T12:39:10.251 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-28T12:39:10.251 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-04-28T12:39:10.267 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout enabled 2024-04-28T12:39:10.283 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout active 2024-04-28T12:39:10.283 INFO:teuthology.orchestra.run.smithi084.stdout:Unit chronyd.service is enabled and running 2024-04-28T12:39:10.283 INFO:teuthology.orchestra.run.smithi084.stdout:Host looks OK 2024-04-28T12:39:10.283 INFO:teuthology.orchestra.run.smithi084.stdout:Cluster fsid: ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:39:10.284 INFO:teuthology.orchestra.run.smithi084.stdout:Acquiring lock 140092121423008 on /run/cephadm/ddee604a-055b-11ef-bc93-c7b262605968.lock 2024-04-28T12:39:10.284 INFO:teuthology.orchestra.run.smithi084.stdout:Lock 140092121423008 acquired on /run/cephadm/ddee604a-055b-11ef-bc93-c7b262605968.lock 2024-04-28T12:39:10.284 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying IP 172.21.15.84 port 3300 ... 2024-04-28T12:39:10.284 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying IP 172.21.15.84 port 6789 ... 2024-04-28T12:39:10.285 INFO:teuthology.orchestra.run.smithi084.stdout:Base mon IP(s) is [172.21.15.84:3300, 172.21.15.84:6789], mon addrv is [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-04-28T12:39:10.292 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.84 metric 100 2024-04-28T12:39:10.292 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.84 metric 100 2024-04-28T12:39:10.299 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-28T12:39:10.299 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-28T12:39:10.299 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1785sec hoplimit 64 pref medium 2024-04-28T12:39:10.305 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-28T12:39:10.305 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-28T12:39:10.305 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-28T12:39:10.306 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-28T12:39:10.306 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d751/64 scope link 2024-04-28T12:39:10.306 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-28T12:39:10.307 INFO:teuthology.orchestra.run.smithi084.stdout:Mon IP `172.21.15.84` is in CIDR network `172.21.0.0/20` 2024-04-28T12:39:10.307 INFO:teuthology.orchestra.run.smithi084.stdout:Mon IP `172.21.15.84` is in CIDR network `172.21.0.0/20` 2024-04-28T12:39:10.307 INFO:teuthology.orchestra.run.smithi084.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-28T12:39:10.307 INFO:teuthology.orchestra.run.smithi084.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-28T12:39:10.308 INFO:teuthology.orchestra.run.smithi084.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T12:39:11.190 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stdout 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-28T12:39:11.190 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-28T12:39:11.191 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Getting image source signatures 2024-04-28T12:39:11.191 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-28T12:39:11.191 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Copying blob sha256:f340d36a00bce3feed0ad509aeafc9092c1c625bf553b8a613c380a79149b87e 2024-04-28T12:39:11.191 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Copying config sha256:55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-28T12:39:11.191 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-28T12:39:12.509 INFO:teuthology.orchestra.run.smithi084.stdout:ceph: stdout ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-28T12:39:12.509 INFO:teuthology.orchestra.run.smithi084.stdout:Ceph version: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-28T12:39:12.509 INFO:teuthology.orchestra.run.smithi084.stdout:Extracting ceph user uid/gid from container image... 2024-04-28T12:39:13.795 INFO:teuthology.orchestra.run.smithi084.stdout:stat: stdout 167 167 2024-04-28T12:39:13.795 INFO:teuthology.orchestra.run.smithi084.stdout:Creating initial keys... 2024-04-28T12:39:15.164 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph-authtool: stdout AQByQy5mKEHGHxAANCaJufxTslT9/57SxJepyg== 2024-04-28T12:39:16.362 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph-authtool: stdout AQBzQy5mGRuDKBAAaSseY1a/VKh6g5FKfESdOg== 2024-04-28T12:39:17.471 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph-authtool: stdout AQB0Qy5mWLcbMhAAZBDVZ5XDLmacuXglKqVBTQ== 2024-04-28T12:39:17.473 INFO:teuthology.orchestra.run.smithi084.stdout:Creating initial monmap... 2024-04-28T12:39:18.710 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-28T12:39:18.711 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-28T12:39:18.711 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:39:18.711 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-28T12:39:18.711 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool for a [v2:172.21.15.84:3300,v1:172.21.15.84:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-28T12:39:18.711 INFO:teuthology.orchestra.run.smithi084.stdout:setting min_mon_release = quincy 2024-04-28T12:39:18.711 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: set fsid to ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:39:18.711 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-28T12:39:18.712 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:39:18.712 INFO:teuthology.orchestra.run.smithi084.stdout:Creating mon... 2024-04-28T12:39:19.808 INFO:teuthology.orchestra.run.smithi084.stdout:create mon.a on 2024-04-28T12:39:20.214 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-28T12:39:20.459 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-28T12:39:20.755 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ddee604a-055b-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-ddee604a-055b-11ef-bc93-c7b262605968.target. 2024-04-28T12:39:20.755 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ddee604a-055b-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-ddee604a-055b-11ef-bc93-c7b262605968.target. 2024-04-28T12:39:21.128 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.a 2024-04-28T12:39:21.128 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-28T12:39:21.421 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ddee604a-055b-11ef-bc93-c7b262605968.target.wants/ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-ddee604a-055b-11ef-bc93-c7b262605968@.service. 2024-04-28T12:39:22.159 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:21 smithi084 podman[27699]: 2024-04-28 12:39:21.887785073 +0000 UTC m=+0.179097273 container create 0ce8f200fe024c92b994003c44fa610dbf29e13fb6865bb9c9061fc89717f688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T12:39:22.159 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:22 smithi084 podman[27699]: 2024-04-28 12:39:22.159234894 +0000 UTC m=+0.450547101 container init 0ce8f200fe024c92b994003c44fa610dbf29e13fb6865bb9c9061fc89717f688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-28T12:39:22.351 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-28T12:39:22.351 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-04-28T12:39:22.371 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-28T12:39:22.371 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-04-28T12:39:22.372 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-04-28T12:39:22.372 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to enable service . firewalld.service is not available 2024-04-28T12:39:22.372 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mon to start... 2024-04-28T12:39:22.373 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mon... 2024-04-28T12:39:22.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:22 smithi084 podman[27699]: 2024-04-28 12:39:22.172037243 +0000 UTC m=+0.463349449 container start 0ce8f200fe024c92b994003c44fa610dbf29e13fb6865bb9c9061fc89717f688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-28T12:39:22.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:22 smithi084 ceph-mon[27717]: mkfs ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:39:22.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:22 smithi084 ceph-mon[27717]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-28T12:39:22.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:22 smithi084 bash[27699]: 0ce8f200fe024c92b994003c44fa610dbf29e13fb6865bb9c9061fc89717f688 2024-04-28T12:39:22.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:22 smithi084 systemd[1]: Started Ceph mon.a for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:39:23.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:23 smithi084 ceph-mon[27717]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-28T12:39:23.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:23 smithi084 ceph-mon[27717]: monmap epoch 1 2024-04-28T12:39:23.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:23 smithi084 ceph-mon[27717]: fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:39:23.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:23 smithi084 ceph-mon[27717]: last_changed 2024-04-28T12:39:18.014667+0000 2024-04-28T12:39:23.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:23 smithi084 ceph-mon[27717]: created 2024-04-28T12:39:18.014667+0000 2024-04-28T12:39:23.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:23 smithi084 ceph-mon[27717]: min_mon_release 19 (squid) 2024-04-28T12:39:23.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:23 smithi084 ceph-mon[27717]: election_strategy: 1 2024-04-28T12:39:23.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:23 smithi084 ceph-mon[27717]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-04-28T12:39:23.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:23 smithi084 ceph-mon[27717]: fsmap 2024-04-28T12:39:23.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:23 smithi084 ceph-mon[27717]: osdmap e1: 0 total, 0 up, 0 in 2024-04-28T12:39:23.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:23 smithi084 ceph-mon[27717]: mgrmap e1: no daemons active 2024-04-28T12:39:23.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:23 smithi084 ceph-mon[27717]: from='client.? 172.21.15.84:0/3510265486' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-28T12:39:23.761 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout cluster: 2024-04-28T12:39:23.761 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout id: ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:39:23.761 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-28T12:39:23.761 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:23.761 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout services: 2024-04-28T12:39:23.761 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.862549s) 2024-04-28T12:39:23.761 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-28T12:39:23.761 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-28T12:39:23.761 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:23.761 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout data: 2024-04-28T12:39:23.762 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-28T12:39:23.762 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-28T12:39:23.762 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-28T12:39:23.762 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout pgs: 2024-04-28T12:39:23.762 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:23.762 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.018+0000 7fa34b000700 1 Processor -- start 2024-04-28T12:39:23.762 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.018+0000 7fa34b000700 1 -- start start 2024-04-28T12:39:23.762 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.018+0000 7fa34b000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 0x7fa3441098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:23.762 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.018+0000 7fa34b000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa344109e80 con 0x7fa344109490 2024-04-28T12:39:23.762 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.019+0000 7fa343e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 0x7fa3441098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:23.762 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.019+0000 7fa343e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 0x7fa3441098b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41086/0 (socket says 172.21.15.84:41086) 2024-04-28T12:39:23.762 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.019+0000 7fa343e00700 1 -- 172.21.15.84:0/970493748 learned_addr learned my addr 172.21.15.84:0/970493748 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.020+0000 7fa343e00700 1 -- 172.21.15.84:0/970493748 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa344106f90 con 0x7fa344109490 2024-04-28T12:39:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.020+0000 7fa343e00700 1 --2- 172.21.15.84:0/970493748 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 0x7fa3441098b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa3340098a0 tx=0x7fa334009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107415142857a64f server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.020+0000 7fa343400700 1 -- 172.21.15.84:0/970493748 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa334004160 con 0x7fa344109490 2024-04-28T12:39:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.020+0000 7fa343400700 1 -- 172.21.15.84:0/970493748 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa334004300 con 0x7fa344109490 2024-04-28T12:39:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.020+0000 7fa343400700 1 -- 172.21.15.84:0/970493748 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa3340036a0 con 0x7fa344109490 2024-04-28T12:39:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.021+0000 7fa34b000700 1 -- 172.21.15.84:0/970493748 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 msgr2=0x7fa3441098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.021+0000 7fa34b000700 1 --2- 172.21.15.84:0/970493748 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 0x7fa3441098b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa3340098a0 tx=0x7fa334009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.021+0000 7fa34b000700 1 -- 172.21.15.84:0/970493748 shutdown_connections 2024-04-28T12:39:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.021+0000 7fa34b000700 1 --2- 172.21.15.84:0/970493748 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 0x7fa3441098b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.021+0000 7fa34b000700 1 -- 172.21.15.84:0/970493748 >> 172.21.15.84:0/970493748 conn(0x7fa344102b30 msgr2=0x7fa344104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.021+0000 7fa34b000700 1 -- 172.21.15.84:0/970493748 shutdown_connections 2024-04-28T12:39:23.763 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.021+0000 7fa34b000700 1 -- 172.21.15.84:0/970493748 wait complete. 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.022+0000 7fa34b000700 1 Processor -- start 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.023+0000 7fa34b000700 1 -- start start 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.023+0000 7fa34b000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 0x7fa34419e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.023+0000 7fa34b000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa344107460 con 0x7fa344109490 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.023+0000 7fa343e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 0x7fa34419e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.023+0000 7fa343e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 0x7fa34419e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41100/0 (socket says 172.21.15.84:41100) 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.023+0000 7fa343e00700 1 -- 172.21.15.84:0/3510265486 learned_addr learned my addr 172.21.15.84:0/3510265486 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.024+0000 7fa343e00700 1 -- 172.21.15.84:0/3510265486 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa34419ed10 con 0x7fa344109490 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.024+0000 7fa343e00700 1 --2- 172.21.15.84:0/3510265486 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 0x7fa34419e7d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa334000c00 tx=0x7fa334004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.024+0000 7fa341600700 1 -- 172.21.15.84:0/3510265486 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa334003db0 con 0x7fa344109490 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.024+0000 7fa341600700 1 -- 172.21.15.84:0/3510265486 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa334011630 con 0x7fa344109490 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.024+0000 7fa34b000700 1 -- 172.21.15.84:0/3510265486 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa34419efa0 con 0x7fa344109490 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.024+0000 7fa341600700 1 -- 172.21.15.84:0/3510265486 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa334011950 con 0x7fa344109490 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.024+0000 7fa34b000700 1 -- 172.21.15.84:0/3510265486 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3441a1c90 con 0x7fa344109490 2024-04-28T12:39:23.764 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.025+0000 7fa341600700 1 -- 172.21.15.84:0/3510265486 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa33401a460 con 0x7fa344109490 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.025+0000 7fa341600700 1 -- 172.21.15.84:0/3510265486 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa33401b780 con 0x7fa344109490 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.025+0000 7fa34b000700 1 -- 172.21.15.84:0/3510265486 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa344064a80 con 0x7fa344109490 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.029+0000 7fa341600700 1 -- 172.21.15.84:0/3510265486 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fa33401b980 con 0x7fa344109490 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.108+0000 7fa34b000700 1 -- 172.21.15.84:0/3510265486 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fa34402e6e0 con 0x7fa344109490 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.108+0000 7fa341600700 1 -- 172.21.15.84:0/3510265486 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fa33401bb60 con 0x7fa344109490 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.109+0000 7fa34b000700 1 -- 172.21.15.84:0/3510265486 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 msgr2=0x7fa34419e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.109+0000 7fa34b000700 1 --2- 172.21.15.84:0/3510265486 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 0x7fa34419e7d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa334000c00 tx=0x7fa334004750 comp rx=0 tx=0).stop 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.109+0000 7fa34b000700 1 -- 172.21.15.84:0/3510265486 shutdown_connections 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.109+0000 7fa34b000700 1 --2- 172.21.15.84:0/3510265486 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa344109490 0x7fa34419e7d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.109+0000 7fa34b000700 1 -- 172.21.15.84:0/3510265486 >> 172.21.15.84:0/3510265486 conn(0x7fa344102b30 msgr2=0x7fa344195560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.110+0000 7fa34b000700 1 -- 172.21.15.84:0/3510265486 shutdown_connections 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:23.110+0000 7fa34b000700 1 -- 172.21.15.84:0/3510265486 wait complete. 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:mon is available 2024-04-28T12:39:23.765 INFO:teuthology.orchestra.run.smithi084.stdout:Assimilating anything we can from ceph.conf... 2024-04-28T12:39:24.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:24 smithi084 ceph-mon[27717]: from='client.? 172.21.15.84:0/3121974147' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-28T12:39:24.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:24 smithi084 ceph-mon[27717]: from='client.? 172.21.15.84:0/3121974147' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-28T12:39:25.124 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:25.124 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [global] 2024-04-28T12:39:25.124 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout fsid = ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:39:25.124 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-28T12:39:25.124 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-04-28T12:39:25.124 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-28T12:39:25.124 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-28T12:39:25.124 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [osd] 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.403+0000 7f2d3f000700 1 Processor -- start 2024-04-28T12:39:25.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.403+0000 7f2d3f000700 1 -- start start 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.404+0000 7f2d3f000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 0x7f2d3810bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.404+0000 7f2d3f000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3807cea0 con 0x7f2d381099c0 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.404+0000 7f2d2fe00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 0x7f2d3810bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.404+0000 7f2d2fe00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 0x7f2d3810bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41112/0 (socket says 172.21.15.84:41112) 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.404+0000 7f2d2fe00700 1 -- 172.21.15.84:0/2276893602 learned_addr learned my addr 172.21.15.84:0/2276893602 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.405+0000 7f2d2fe00700 1 -- 172.21.15.84:0/2276893602 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3810c2f0 con 0x7f2d381099c0 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.405+0000 7f2d2fe00700 1 --2- 172.21.15.84:0/2276893602 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 0x7f2d3810bdb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2d280098a0 tx=0x7f2d28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7ae49e7a8369333 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.405+0000 7f2d37400700 1 -- 172.21.15.84:0/2276893602 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d28004160 con 0x7f2d381099c0 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.405+0000 7f2d37400700 1 -- 172.21.15.84:0/2276893602 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d28004300 con 0x7f2d381099c0 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.405+0000 7f2d37400700 1 -- 172.21.15.84:0/2276893602 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d280036a0 con 0x7f2d381099c0 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.406+0000 7f2d3f000700 1 -- 172.21.15.84:0/2276893602 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 msgr2=0x7f2d3810bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.406+0000 7f2d3f000700 1 --2- 172.21.15.84:0/2276893602 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 0x7f2d3810bdb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2d280098a0 tx=0x7f2d28009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.407+0000 7f2d3f000700 1 -- 172.21.15.84:0/2276893602 shutdown_connections 2024-04-28T12:39:25.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.407+0000 7f2d3f000700 1 --2- 172.21.15.84:0/2276893602 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 0x7f2d3810bdb0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.407+0000 7f2d3f000700 1 -- 172.21.15.84:0/2276893602 >> 172.21.15.84:0/2276893602 conn(0x7f2d38103410 msgr2=0x7f2d38105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.407+0000 7f2d3f000700 1 -- 172.21.15.84:0/2276893602 shutdown_connections 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.407+0000 7f2d3f000700 1 -- 172.21.15.84:0/2276893602 wait complete. 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.408+0000 7f2d3f000700 1 Processor -- start 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.409+0000 7f2d3f000700 1 -- start start 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.409+0000 7f2d3f000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 0x7f2d38195330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.409+0000 7f2d3f000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3810c7c0 con 0x7f2d381099c0 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.409+0000 7f2d2fe00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 0x7f2d38195330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.409+0000 7f2d2fe00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 0x7f2d38195330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41122/0 (socket says 172.21.15.84:41122) 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.409+0000 7f2d2fe00700 1 -- 172.21.15.84:0/3121974147 learned_addr learned my addr 172.21.15.84:0/3121974147 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.409+0000 7f2d2fe00700 1 -- 172.21.15.84:0/3121974147 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d38195870 con 0x7f2d381099c0 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.410+0000 7f2d2fe00700 1 --2- 172.21.15.84:0/3121974147 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 0x7f2d38195330 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2d28000c00 tx=0x7f2d28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.410+0000 7f2d35600700 1 -- 172.21.15.84:0/3121974147 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d28003db0 con 0x7f2d381099c0 2024-04-28T12:39:25.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.410+0000 7f2d35600700 1 -- 172.21.15.84:0/3121974147 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d28011630 con 0x7f2d381099c0 2024-04-28T12:39:25.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.410+0000 7f2d35600700 1 -- 172.21.15.84:0/3121974147 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d28011950 con 0x7f2d381099c0 2024-04-28T12:39:25.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.410+0000 7f2d3f000700 1 -- 172.21.15.84:0/3121974147 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d38195b00 con 0x7f2d381099c0 2024-04-28T12:39:25.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.410+0000 7f2d3f000700 1 -- 172.21.15.84:0/3121974147 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d381920c0 con 0x7f2d381099c0 2024-04-28T12:39:25.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.411+0000 7f2d35600700 1 -- 172.21.15.84:0/3121974147 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2d28011af0 con 0x7f2d381099c0 2024-04-28T12:39:25.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.411+0000 7f2d35600700 1 -- 172.21.15.84:0/3121974147 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2d2801c450 con 0x7f2d381099c0 2024-04-28T12:39:25.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.411+0000 7f2d3f000700 1 -- 172.21.15.84:0/3121974147 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d38195cd0 con 0x7f2d381099c0 2024-04-28T12:39:25.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.414+0000 7f2d35600700 1 -- 172.21.15.84:0/3121974147 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f2d38195cd0 con 0x7f2d381099c0 2024-04-28T12:39:25.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.491+0000 7f2d3f000700 1 -- 172.21.15.84:0/3121974147 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f2d381981c0 con 0x7f2d381099c0 2024-04-28T12:39:25.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.499+0000 7f2d35600700 1 -- 172.21.15.84:0/3121974147 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+490 (secure 0 0 0) 0x7f2d28032020 con 0x7f2d381099c0 2024-04-28T12:39:25.129 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.499+0000 7f2d35600700 1 -- 172.21.15.84:0/3121974147 <== mon.0 v2:172.21.15.84:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2d28004260 con 0x7f2d381099c0 2024-04-28T12:39:25.129 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.500+0000 7f2d3f000700 1 -- 172.21.15.84:0/3121974147 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 msgr2=0x7f2d38195330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:25.129 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.500+0000 7f2d3f000700 1 --2- 172.21.15.84:0/3121974147 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 0x7f2d38195330 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2d28000c00 tx=0x7f2d28004750 comp rx=0 tx=0).stop 2024-04-28T12:39:25.129 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.500+0000 7f2d3f000700 1 -- 172.21.15.84:0/3121974147 shutdown_connections 2024-04-28T12:39:25.129 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.500+0000 7f2d3f000700 1 --2- 172.21.15.84:0/3121974147 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d381099c0 0x7f2d38195330 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:25.129 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.500+0000 7f2d3f000700 1 -- 172.21.15.84:0/3121974147 >> 172.21.15.84:0/3121974147 conn(0x7f2d38103410 msgr2=0x7f2d38105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:25.130 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.501+0000 7f2d3f000700 1 -- 172.21.15.84:0/3121974147 shutdown_connections 2024-04-28T12:39:25.130 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:24.501+0000 7f2d3f000700 1 -- 172.21.15.84:0/3121974147 wait complete. 2024-04-28T12:39:25.130 INFO:teuthology.orchestra.run.smithi084.stdout:Generating new minimal ceph.conf... 2024-04-28T12:39:26.085 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:25 smithi084 ceph-mon[27717]: from='client.? 172.21.15.84:0/596907280' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:39:26.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.684+0000 7f91fdc00700 1 Processor -- start 2024-04-28T12:39:26.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.684+0000 7f91fdc00700 1 -- start start 2024-04-28T12:39:26.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.685+0000 7f91fdc00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 0x7f91f810be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:26.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.685+0000 7f91fdc00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f91f807cde0 con 0x7f91f8109a80 2024-04-28T12:39:26.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.685+0000 7f91f6a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 0x7f91f810be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:26.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.685+0000 7f91f6a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 0x7f91f810be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41134/0 (socket says 172.21.15.84:41134) 2024-04-28T12:39:26.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.685+0000 7f91f6a00700 1 -- 172.21.15.84:0/3162654782 learned_addr learned my addr 172.21.15.84:0/3162654782 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:26.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.686+0000 7f91f6a00700 1 -- 172.21.15.84:0/3162654782 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f810c3b0 con 0x7f91f8109a80 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.686+0000 7f91f6a00700 1 --2- 172.21.15.84:0/3162654782 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 0x7f91f810be70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f91e80098a0 tx=0x7f91e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2ad2a82e2d26966 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.686+0000 7f91f5600700 1 -- 172.21.15.84:0/3162654782 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91e8004160 con 0x7f91f8109a80 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.686+0000 7f91f5600700 1 -- 172.21.15.84:0/3162654782 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f91e8004300 con 0x7f91f8109a80 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.686+0000 7f91f5600700 1 -- 172.21.15.84:0/3162654782 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91e8011af0 con 0x7f91f8109a80 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.687+0000 7f91fdc00700 1 -- 172.21.15.84:0/3162654782 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 msgr2=0x7f91f810be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.687+0000 7f91fdc00700 1 --2- 172.21.15.84:0/3162654782 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 0x7f91f810be70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f91e80098a0 tx=0x7f91e8009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.687+0000 7f91fdc00700 1 -- 172.21.15.84:0/3162654782 shutdown_connections 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.687+0000 7f91fdc00700 1 --2- 172.21.15.84:0/3162654782 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 0x7f91f810be70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.687+0000 7f91fdc00700 1 -- 172.21.15.84:0/3162654782 >> 172.21.15.84:0/3162654782 conn(0x7f91f8103410 msgr2=0x7f91f8105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.687+0000 7f91fdc00700 1 -- 172.21.15.84:0/3162654782 shutdown_connections 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.687+0000 7f91fdc00700 1 -- 172.21.15.84:0/3162654782 wait complete. 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.688+0000 7f91fdc00700 1 Processor -- start 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.689+0000 7f91fdc00700 1 -- start start 2024-04-28T12:39:26.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.689+0000 7f91fdc00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 0x7f91f819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.689+0000 7f91fdc00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f91f810d250 con 0x7f91f8109a80 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.689+0000 7f91f6a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 0x7f91f819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.689+0000 7f91f6a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 0x7f91f819f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41142/0 (socket says 172.21.15.84:41142) 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.689+0000 7f91f6a00700 1 -- 172.21.15.84:0/596907280 learned_addr learned my addr 172.21.15.84:0/596907280 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.690+0000 7f91f6a00700 1 -- 172.21.15.84:0/596907280 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f819f5c0 con 0x7f91f8109a80 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.690+0000 7f91f6a00700 1 --2- 172.21.15.84:0/596907280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 0x7f91f819f080 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f91e8011ac0 tx=0x7f91e8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.690+0000 7f91ef400700 1 -- 172.21.15.84:0/596907280 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91e8003b20 con 0x7f91f8109a80 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.690+0000 7f91ef400700 1 -- 172.21.15.84:0/596907280 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f91e801a8c0 con 0x7f91f8109a80 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.690+0000 7f91fdc00700 1 -- 172.21.15.84:0/596907280 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f819f850 con 0x7f91f8109a80 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.690+0000 7f91ef400700 1 -- 172.21.15.84:0/596907280 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91e80248d0 con 0x7f91f8109a80 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.690+0000 7f91fdc00700 1 -- 172.21.15.84:0/596907280 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f81a2540 con 0x7f91f8109a80 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.691+0000 7f91ef400700 1 -- 172.21.15.84:0/596907280 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f91e8022020 con 0x7f91f8109a80 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.691+0000 7f91ef400700 1 -- 172.21.15.84:0/596907280 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f91e802d9d0 con 0x7f91f8109a80 2024-04-28T12:39:26.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.692+0000 7f91fdc00700 1 -- 172.21.15.84:0/596907280 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91f8064a80 con 0x7f91f8109a80 2024-04-28T12:39:26.414 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.695+0000 7f91ef400700 1 -- 172.21.15.84:0/596907280 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f91e802d3f0 con 0x7f91f8109a80 2024-04-28T12:39:26.414 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.770+0000 7f91fdc00700 1 -- 172.21.15.84:0/596907280 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f91f81a2a80 con 0x7f91f8109a80 2024-04-28T12:39:26.414 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.771+0000 7f91ef400700 1 -- 172.21.15.84:0/596907280 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f91e801aa70 con 0x7f91f8109a80 2024-04-28T12:39:26.414 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.772+0000 7f91fdc00700 1 -- 172.21.15.84:0/596907280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 msgr2=0x7f91f819f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:26.414 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.772+0000 7f91fdc00700 1 --2- 172.21.15.84:0/596907280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 0x7f91f819f080 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f91e8011ac0 tx=0x7f91e8011ba0 comp rx=0 tx=0).stop 2024-04-28T12:39:26.414 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.773+0000 7f91fdc00700 1 -- 172.21.15.84:0/596907280 shutdown_connections 2024-04-28T12:39:26.414 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.773+0000 7f91fdc00700 1 --2- 172.21.15.84:0/596907280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f8109a80 0x7f91f819f080 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:26.414 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.773+0000 7f91fdc00700 1 -- 172.21.15.84:0/596907280 >> 172.21.15.84:0/596907280 conn(0x7f91f8103410 msgr2=0x7f91f8103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:26.414 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.773+0000 7f91fdc00700 1 -- 172.21.15.84:0/596907280 shutdown_connections 2024-04-28T12:39:26.414 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:25.773+0000 7f91fdc00700 1 -- 172.21.15.84:0/596907280 wait complete. 2024-04-28T12:39:26.415 INFO:teuthology.orchestra.run.smithi084.stdout:Restarting the monitor... 2024-04-28T12:39:26.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:26 smithi084 systemd[1]: Stopping Ceph mon.a for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:39:27.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:26 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a[27713]: 2024-04-28T12:39:26.893+0000 7f7941200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T12:39:27.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:26 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a[27713]: 2024-04-28T12:39:26.893+0000 7f7941200700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-28T12:39:27.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:27 smithi084 podman[27958]: 2024-04-28 12:39:27.059082861 +0000 UTC m=+0.300580704 container died 0ce8f200fe024c92b994003c44fa610dbf29e13fb6865bb9c9061fc89717f688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-28T12:39:27.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:27 smithi084 podman[27958]: 2024-04-28 12:39:27.529463206 +0000 UTC m=+0.770961079 container cleanup 0ce8f200fe024c92b994003c44fa610dbf29e13fb6865bb9c9061fc89717f688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T12:39:27.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:27 smithi084 bash[27958]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a 2024-04-28T12:39:28.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:27 smithi084 podman[27972]: 2024-04-28 12:39:27.763482285 +0000 UTC m=+0.685175740 container remove 0ce8f200fe024c92b994003c44fa610dbf29e13fb6865bb9c9061fc89717f688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-28T12:39:28.621 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 systemd[1]: ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-28T12:39:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 systemd[1]: Stopped Ceph mon.a for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:39:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 systemd[1]: ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.a.service: Consumed 1.938s CPU time. 2024-04-28T12:39:28.622 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 systemd[1]: Starting Ceph mon.a for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:39:28.906 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 podman[28062]: 2024-04-28 12:39:28.529992554 +0000 UTC m=+0.035920548 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:39:28.906 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 podman[28062]: 2024-04-28 12:39:28.680926843 +0000 UTC m=+0.186854899 container create d77f816c8b1cc022e5e3fb02a1895259519bb9733db38fde60ffd0b21ccac878 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-28T12:39:28.906 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 podman[28062]: 2024-04-28 12:39:28.909484637 +0000 UTC m=+0.415412673 container init d77f816c8b1cc022e5e3fb02a1895259519bb9733db38fde60ffd0b21ccac878 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T12:39:29.006 INFO:teuthology.orchestra.run.smithi084.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-28T12:39:29.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 podman[28062]: 2024-04-28 12:39:28.919229378 +0000 UTC m=+0.425157437 container start d77f816c8b1cc022e5e3fb02a1895259519bb9733db38fde60ffd0b21ccac878 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-28T12:39:29.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: set uid:gid to 167:167 (ceph:ceph) 2024-04-28T12:39:29.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-28T12:39:29.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: pidfile_write: ignore empty --pid-file 2024-04-28T12:39:29.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: load: jerasure load: lrc 2024-04-28T12:39:29.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: RocksDB version: 7.9.2 2024-04-28T12:39:29.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Git sha 0 2024-04-28T12:39:29.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: DB SUMMARY 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: DB Session ID: 6LGHBBNCLVWV21EKXJ9D 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: CURRENT file: CURRENT 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: IDENTITY file: IDENTITY 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105684 ; 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.error_if_exists: 0 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.create_if_missing: 0 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.paranoid_checks: 1 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-28T12:39:29.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-28T12:39:29.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.env: 0x558b5068ccc0 2024-04-28T12:39:29.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.fs: PosixFileSystem 2024-04-28T12:39:29.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.info_log: 0x558b72ab3dc0 2024-04-28T12:39:29.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_file_opening_threads: 16 2024-04-28T12:39:29.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.statistics: (nil) 2024-04-28T12:39:29.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.use_fsync: 0 2024-04-28T12:39:29.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_log_file_size: 0 2024-04-28T12:39:29.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-28T12:39:29.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-28T12:39:29.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.keep_log_file_num: 1000 2024-04-28T12:39:29.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.recycle_log_file_num: 0 2024-04-28T12:39:29.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.allow_fallocate: 1 2024-04-28T12:39:29.172 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.allow_mmap_reads: 0 2024-04-28T12:39:29.172 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.allow_mmap_writes: 0 2024-04-28T12:39:29.172 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.use_direct_reads: 0 2024-04-28T12:39:29.172 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-28T12:39:29.172 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.create_missing_column_families: 0 2024-04-28T12:39:29.172 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.db_log_dir: 2024-04-28T12:39:29.172 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.wal_dir: 2024-04-28T12:39:29.172 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.advise_random_on_open: 1 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.db_write_buffer_size: 0 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.write_buffer_manager: 0x558b72a850e0 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.rate_limiter: (nil) 2024-04-28T12:39:29.173 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.wal_recovery_mode: 2 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.enable_thread_tracking: 0 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.enable_pipelined_write: 0 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.unordered_write: 0 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.row_cache: None 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.wal_filter: None 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.allow_ingest_behind: 0 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.two_write_queues: 0 2024-04-28T12:39:29.174 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.manual_wal_flush: 0 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.wal_compression: 0 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.atomic_flush: 0 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.log_readahead_size: 0 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.best_efforts_recovery: 0 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.allow_data_in_errors: 0 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.db_host_id: __hostname__ 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-28T12:39:29.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_background_jobs: 2 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_background_compactions: -1 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_subcompactions: 1 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_total_wal_size: 0 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_open_files: -1 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bytes_per_sync: 0 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-28T12:39:29.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_readahead_size: 0 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_background_flushes: -1 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Compression algorithms supported: 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: kZSTD supported: 0 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: kXpressCompression supported: 0 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: kLZ4HCCompression supported: 1 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: kZlibCompression supported: 1 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: kSnappyCompression supported: 1 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: kLZ4Compression supported: 1 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: kBZip2Compression supported: 0 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-28T12:39:29.177 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.merge_operator: 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_filter: None 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_filter_factory: None 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.sst_partitioner_factory: None 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558b72ab3f80) 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout: cache_index_and_filter_blocks: 1 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout: pin_top_level_index_and_filter: 1 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout: index_type: 0 2024-04-28T12:39:29.178 INFO:journalctl@ceph.mon.a.smithi084.stdout: data_block_index_type: 0 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: index_shortening: 1 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: checksum: 4 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: no_block_cache: 0 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache: 0x558b72ab71f0 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache_name: BinnedLRUCache 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache_options: 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: capacity : 536870912 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: num_shard_bits : 4 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: strict_capacity_limit : 0 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: high_pri_pool_ratio: 0.000 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache_compressed: (nil) 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: persistent_cache: (nil) 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_size: 4096 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_size_deviation: 10 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_restart_interval: 16 2024-04-28T12:39:29.179 INFO:journalctl@ceph.mon.a.smithi084.stdout: index_block_restart_interval: 1 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: metadata_block_size: 4096 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: partition_filters: 0 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: use_delta_encoding: 1 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: filter_policy: bloomfilter 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: whole_key_filtering: 1 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: verify_compression: 0 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: read_amp_bytes_per_bit: 0 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: format_version: 5 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: enable_index_compression: 1 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_align: 0 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: max_auto_readahead_size: 262144 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: prepopulate_block_cache: 0 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: initial_auto_readahead_size: 8192 2024-04-28T12:39:29.180 INFO:journalctl@ceph.mon.a.smithi084.stdout: num_file_reads_for_auto_readahead: 2 2024-04-28T12:39:29.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.write_buffer_size: 33554432 2024-04-28T12:39:29.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_write_buffer_number: 2 2024-04-28T12:39:29.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compression: NoCompression 2024-04-28T12:39:29.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bottommost_compression: Disabled 2024-04-28T12:39:29.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.prefix_extractor: nullptr 2024-04-28T12:39:29.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-28T12:39:29.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.num_levels: 7 2024-04-28T12:39:29.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-28T12:39:29.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-28T12:39:29.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-28T12:39:29.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-28T12:39:29.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-28T12:39:29.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-28T12:39:29.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-28T12:39:29.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-28T12:39:29.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-28T12:39:29.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-28T12:39:29.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compression_opts.level: 32767 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compression_opts.strategy: 0 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compression_opts.enabled: false 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-28T12:39:29.183 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.target_file_size_base: 67108864 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-28T12:39:29.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.arena_block_size: 1048576 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.disable_auto_compactions: 0 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-28T12:39:29.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.inplace_update_support: 0 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.bloom_locality: 0 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.max_successive_merges: 0 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.paranoid_file_checks: 0 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.force_consistency_checks: 1 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.report_bg_io_stats: 0 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.ttl: 2592000 2024-04-28T12:39:29.186 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.enable_blob_files: false 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.min_blob_size: 0 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.blob_file_size: 268435456 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.blob_file_starting_level: 0 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-28T12:39:29.187 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: bf83acbb-ec7c-421b-9b89-dd98b606a12e 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714307968975087, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714307968977258, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100780, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98678, "index_size": 294, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14308, "raw_average_key_size": 49, "raw_value_size": 90891, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714307968, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bf83acbb-ec7c-421b-9b89-dd98b606a12e", "db_session_id": "6LGHBBNCLVWV21EKXJ9D", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714307968977427, "job": 1, "event": "recovery_finished"} 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558b72bd4000 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: DB pointer 0x558b72bc0000 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** DB Stats ** 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-28T12:39:29.188 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** Compaction Stats [default] ** 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: L0 2/0 100.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: Sum 2/0 100.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** Compaction Stats [default] ** 2024-04-28T12:39:29.189 INFO:journalctl@ceph.mon.a.smithi084.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 65.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative compaction: 0.00 GB write, 14.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval compaction: 0.00 GB write, 14.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-28T12:39:29.190 INFO:journalctl@ceph.mon.a.smithi084.stdout: Block cache BinnedLRUCache@0x558b72ab71f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: starting mon.a rank 0 at public addrs [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] at bind addrs [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: mon.a@-1(???) e1 preinit fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: mon.a@-1(???).mds e1 new map 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: mon.a@-1(???).mds e1 print_map 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout: e1 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout: legacy client fscid: -1 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout: No filesystems configured 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-28T12:39:29.191 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:28 smithi084 ceph-mon[28081]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 bash[28062]: d77f816c8b1cc022e5e3fb02a1895259519bb9733db38fde60ffd0b21ccac878 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 systemd[1]: Started Ceph mon.a for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 ceph-mon[28081]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 ceph-mon[28081]: monmap epoch 1 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 ceph-mon[28081]: fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 ceph-mon[28081]: last_changed 2024-04-28T12:39:18.014667+0000 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 ceph-mon[28081]: created 2024-04-28T12:39:18.014667+0000 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 ceph-mon[28081]: min_mon_release 19 (squid) 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 ceph-mon[28081]: election_strategy: 1 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 ceph-mon[28081]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 ceph-mon[28081]: fsmap 2024-04-28T12:39:29.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 ceph-mon[28081]: osdmap e1: 0 total, 0 up, 0 in 2024-04-28T12:39:29.193 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:29 smithi084 ceph-mon[28081]: mgrmap e1: no daemons active 2024-04-28T12:39:30.430 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.741+0000 7f8546200700 1 Processor -- start 2024-04-28T12:39:30.430 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.741+0000 7f8546200700 1 -- start start 2024-04-28T12:39:30.431 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.742+0000 7f8546200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 0x7f854010bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:30.431 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.742+0000 7f8546200700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f854007ce40 con 0x7f85401099f0 2024-04-28T12:39:30.431 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.743+0000 7f853f400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 0x7f854010bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:30.433 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.743+0000 7f853f400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 0x7f854010bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:44508/0 (socket says 172.21.15.84:44508) 2024-04-28T12:39:30.433 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.743+0000 7f853f400700 1 -- 172.21.15.84:0/609647904 learned_addr learned my addr 172.21.15.84:0/609647904 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:30.433 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.744+0000 7f853f400700 1 -- 172.21.15.84:0/609647904 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854010c320 con 0x7f85401099f0 2024-04-28T12:39:30.433 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.744+0000 7f853f400700 1 --2- 172.21.15.84:0/609647904 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 0x7f854010bde0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f85300098a0 tx=0x7f8530009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf41bafa8bc24c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:30.433 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.744+0000 7f853e000700 1 -- 172.21.15.84:0/609647904 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8530004160 con 0x7f85401099f0 2024-04-28T12:39:30.433 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.744+0000 7f853e000700 1 -- 172.21.15.84:0/609647904 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8530004300 con 0x7f85401099f0 2024-04-28T12:39:30.433 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.744+0000 7f853e000700 1 -- 172.21.15.84:0/609647904 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8530011af0 con 0x7f85401099f0 2024-04-28T12:39:30.433 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.745+0000 7f8546200700 1 -- 172.21.15.84:0/609647904 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 msgr2=0x7f854010bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.745+0000 7f8546200700 1 --2- 172.21.15.84:0/609647904 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 0x7f854010bde0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f85300098a0 tx=0x7f8530009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.745+0000 7f8546200700 1 -- 172.21.15.84:0/609647904 shutdown_connections 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.745+0000 7f8546200700 1 --2- 172.21.15.84:0/609647904 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 0x7f854010bde0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.745+0000 7f8546200700 1 -- 172.21.15.84:0/609647904 >> 172.21.15.84:0/609647904 conn(0x7f85401033a0 msgr2=0x7f8540105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.745+0000 7f8546200700 1 -- 172.21.15.84:0/609647904 shutdown_connections 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.745+0000 7f8546200700 1 -- 172.21.15.84:0/609647904 wait complete. 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.746+0000 7f8546200700 1 Processor -- start 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.746+0000 7f8546200700 1 -- start start 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.747+0000 7f8546200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 0x7f854019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.747+0000 7f8546200700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f854010d290 con 0x7f85401099f0 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.747+0000 7f853f400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 0x7f854019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.747+0000 7f853f400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 0x7f854019ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:44522/0 (socket says 172.21.15.84:44522) 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.747+0000 7f853f400700 1 -- 172.21.15.84:0/602427106 learned_addr learned my addr 172.21.15.84:0/602427106 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:30.434 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.748+0000 7f853f400700 1 -- 172.21.15.84:0/602427106 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854019d3d0 con 0x7f85401099f0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.748+0000 7f853f400700 1 --2- 172.21.15.84:0/602427106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 0x7f854019ce90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8530009fb0 tx=0x7f8530011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.748+0000 7f8537e00700 1 -- 172.21.15.84:0/602427106 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85300036a0 con 0x7f85401099f0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.748+0000 7f8537e00700 1 -- 172.21.15.84:0/602427106 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f853001a430 con 0x7f85401099f0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.748+0000 7f8537e00700 1 -- 172.21.15.84:0/602427106 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8530019550 con 0x7f85401099f0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.748+0000 7f8546200700 1 -- 172.21.15.84:0/602427106 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f854019d660 con 0x7f85401099f0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.748+0000 7f8546200700 1 -- 172.21.15.84:0/602427106 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85401a0350 con 0x7f85401099f0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.749+0000 7f8537e00700 1 -- 172.21.15.84:0/602427106 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8530019b90 con 0x7f85401099f0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.749+0000 7f8537e00700 1 -- 172.21.15.84:0/602427106 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8530024730 con 0x7f85401099f0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.749+0000 7f8546200700 1 -- 172.21.15.84:0/602427106 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8540064a80 con 0x7f85401099f0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.753+0000 7f8537e00700 1 -- 172.21.15.84:0/602427106 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f8530043ec0 con 0x7f85401099f0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.831+0000 7f8546200700 1 -- 172.21.15.84:0/602427106 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f85401a0520 con 0x7f85401099f0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.835+0000 7f8537e00700 1 -- 172.21.15.84:0/602427106 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f8530032020 con 0x7f85401099f0 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.836+0000 7f8546200700 1 -- 172.21.15.84:0/602427106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 msgr2=0x7f854019ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.836+0000 7f8546200700 1 --2- 172.21.15.84:0/602427106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 0x7f854019ce90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8530009fb0 tx=0x7f8530011940 comp rx=0 tx=0).stop 2024-04-28T12:39:30.435 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.837+0000 7f8546200700 1 -- 172.21.15.84:0/602427106 shutdown_connections 2024-04-28T12:39:30.436 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.837+0000 7f8546200700 1 --2- 172.21.15.84:0/602427106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85401099f0 0x7f854019ce90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:30.436 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.837+0000 7f8546200700 1 -- 172.21.15.84:0/602427106 >> 172.21.15.84:0/602427106 conn(0x7f85401033a0 msgr2=0x7f8540103f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:30.436 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.837+0000 7f8546200700 1 -- 172.21.15.84:0/602427106 shutdown_connections 2024-04-28T12:39:30.436 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:29.837+0000 7f8546200700 1 -- 172.21.15.84:0/602427106 wait complete. 2024-04-28T12:39:30.436 INFO:teuthology.orchestra.run.smithi084.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-28T12:39:30.454 INFO:teuthology.orchestra.run.smithi084.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-28T12:39:30.454 INFO:teuthology.orchestra.run.smithi084.stdout:Creating mgr... 2024-04-28T12:39:30.455 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-28T12:39:30.456 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-28T12:39:30.877 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.x 2024-04-28T12:39:30.878 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-28T12:39:31.128 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ddee604a-055b-11ef-bc93-c7b262605968.target.wants/ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-ddee604a-055b-11ef-bc93-c7b262605968@.service. 2024-04-28T12:39:31.159 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:30 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/602427106' entity='client.admin' 2024-04-28T12:39:31.417 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:31 smithi084 systemd[1]: Starting Ceph mgr.x for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:39:31.824 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:31 smithi084 podman[28294]: 2024-04-28 12:39:31.430093874 +0000 UTC m=+0.033459335 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-28T12:39:31.824 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:31 smithi084 podman[28294]: 2024-04-28 12:39:31.543799712 +0000 UTC m=+0.147165207 container create 9ede3f8332e62dc302a924e39dd54ce9e7a4074dedaf96e7ccae3427f83ab661 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424) 2024-04-28T12:39:31.825 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:31 smithi084 podman[28294]: 2024-04-28 12:39:31.763579276 +0000 UTC m=+0.366944781 container init 9ede3f8332e62dc302a924e39dd54ce9e7a4074dedaf96e7ccae3427f83ab661 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-28T12:39:31.825 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:31 smithi084 podman[28294]: 2024-04-28 12:39:31.772822601 +0000 UTC m=+0.376188083 container start 9ede3f8332e62dc302a924e39dd54ce9e7a4074dedaf96e7ccae3427f83ab661 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-28T12:39:31.825 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:31 smithi084 bash[28294]: 9ede3f8332e62dc302a924e39dd54ce9e7a4074dedaf96e7ccae3427f83ab661 2024-04-28T12:39:31.854 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-28T12:39:31.854 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-04-28T12:39:31.869 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-28T12:39:31.869 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-04-28T12:39:31.869 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-04-28T12:39:31.869 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to enable service . firewalld.service is not available 2024-04-28T12:39:31.888 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-28T12:39:31.889 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-04-28T12:39:31.904 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-28T12:39:31.904 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-04-28T12:39:31.904 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-04-28T12:39:31.904 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-28T12:39:31.904 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr to start... 2024-04-28T12:39:31.904 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr... 2024-04-28T12:39:32.101 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:31 smithi084 systemd[1]: Started Ceph mgr.x for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:39:32.101 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:32 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:32.012+0000 7feb1a348200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T12:39:32.388 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:32 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:32.276+0000 7feb1a348200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:39:32.648 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:32 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:32.407+0000 7feb1a348200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T12:39:32.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:32 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1473745491' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-28T12:39:32.917 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:32 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:32.817+0000 7feb1a348200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T12:39:33.280 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:32 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:32.928+0000 7feb1a348200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T12:39:33.291 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:33.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:39:33.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "ddee604a-055b-11ef-bc93-c7b262605968", 2024-04-28T12:39:33.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-04-28T12:39:33.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-28T12:39:33.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-28T12:39:33.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-28T12:39:33.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-28T12:39:33.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-28T12:39:33.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-28T12:39:33.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-28T12:39:33.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-28T12:39:33.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-04-28T12:39:33.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:39:33.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:39:33.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:33.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-28T12:39:33.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:33.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-04-28T12:39:22.249243+0000", 2024-04-28T12:39:33.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:39:33.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:33.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.544+0000 7f33ea600700 1 Processor -- start 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.544+0000 7f33ea600700 1 -- start start 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.544+0000 7f33ea600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 0x7f33e407d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.544+0000 7f33ea600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f33e407d810 con 0x7f33e407ee20 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.545+0000 7f33e3400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 0x7f33e407d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.545+0000 7f33e3400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 0x7f33e407d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:44556/0 (socket says 172.21.15.84:44556) 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.545+0000 7f33e3400700 1 -- 172.21.15.84:0/3261924494 learned_addr learned my addr 172.21.15.84:0/3261924494 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.545+0000 7f33e3400700 1 -- 172.21.15.84:0/3261924494 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e407d990 con 0x7f33e407ee20 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.546+0000 7f33e3400700 1 --2- 172.21.15.84:0/3261924494 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 0x7f33e407d2d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f33d40098a0 tx=0x7f33d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82858a32108b7fa server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.546+0000 7f33e2000700 1 -- 172.21.15.84:0/3261924494 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33d4004160 con 0x7f33e407ee20 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.546+0000 7f33e2000700 1 -- 172.21.15.84:0/3261924494 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f33d4004300 con 0x7f33e407ee20 2024-04-28T12:39:33.307 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.546+0000 7f33e2000700 1 -- 172.21.15.84:0/3261924494 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33d4011af0 con 0x7f33e407ee20 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.547+0000 7f33ea600700 1 -- 172.21.15.84:0/3261924494 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 msgr2=0x7f33e407d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.547+0000 7f33ea600700 1 --2- 172.21.15.84:0/3261924494 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 0x7f33e407d2d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f33d40098a0 tx=0x7f33d4009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.547+0000 7f33ea600700 1 -- 172.21.15.84:0/3261924494 shutdown_connections 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.547+0000 7f33ea600700 1 --2- 172.21.15.84:0/3261924494 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 0x7f33e407d2d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.547+0000 7f33ea600700 1 -- 172.21.15.84:0/3261924494 >> 172.21.15.84:0/3261924494 conn(0x7f33e4104470 msgr2=0x7f33e41068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.547+0000 7f33ea600700 1 -- 172.21.15.84:0/3261924494 shutdown_connections 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.548+0000 7f33ea600700 1 -- 172.21.15.84:0/3261924494 wait complete. 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.549+0000 7f33ea600700 1 Processor -- start 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.549+0000 7f33ea600700 1 -- start start 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.549+0000 7f33ea600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 0x7f33e419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.549+0000 7f33ea600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f33e410f480 con 0x7f33e407ee20 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.549+0000 7f33e3400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 0x7f33e419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.549+0000 7f33e3400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 0x7f33e419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:44568/0 (socket says 172.21.15.84:44568) 2024-04-28T12:39:33.308 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.549+0000 7f33e3400700 1 -- 172.21.15.84:0/1473745491 learned_addr learned my addr 172.21.15.84:0/1473745491 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.550+0000 7f33e3400700 1 -- 172.21.15.84:0/1473745491 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e419f5d0 con 0x7f33e407ee20 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.550+0000 7f33e3400700 1 --2- 172.21.15.84:0/1473745491 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 0x7f33e419f090 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f33d4011ac0 tx=0x7f33d4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.550+0000 7f33dbe00700 1 -- 172.21.15.84:0/1473745491 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33d4003970 con 0x7f33e407ee20 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.550+0000 7f33dbe00700 1 -- 172.21.15.84:0/1473745491 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f33d4003b10 con 0x7f33e407ee20 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.550+0000 7f33ea600700 1 -- 172.21.15.84:0/1473745491 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33e419f860 con 0x7f33e407ee20 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.550+0000 7f33dbe00700 1 -- 172.21.15.84:0/1473745491 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33d4024610 con 0x7f33e407ee20 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.550+0000 7f33ea600700 1 -- 172.21.15.84:0/1473745491 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33e41a2550 con 0x7f33e407ee20 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.551+0000 7f33dbe00700 1 -- 172.21.15.84:0/1473745491 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f33d4024bd0 con 0x7f33e407ee20 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.551+0000 7f33dbe00700 1 -- 172.21.15.84:0/1473745491 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f33d4019700 con 0x7f33e407ee20 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.551+0000 7f33ea600700 1 -- 172.21.15.84:0/1473745491 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33e4064a80 con 0x7f33e407ee20 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.554+0000 7f33dbe00700 1 -- 172.21.15.84:0/1473745491 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f33d4043ec0 con 0x7f33e407ee20 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.633+0000 7f33ea600700 1 -- 172.21.15.84:0/1473745491 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f33e41a2720 con 0x7f33e407ee20 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.634+0000 7f33dbe00700 1 -- 172.21.15.84:0/1473745491 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f33d4032030 con 0x7f33e407ee20 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.635+0000 7f33ea600700 1 -- 172.21.15.84:0/1473745491 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 msgr2=0x7f33e419f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:33.309 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.635+0000 7f33ea600700 1 --2- 172.21.15.84:0/1473745491 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 0x7f33e419f090 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f33d4011ac0 tx=0x7f33d4011ba0 comp rx=0 tx=0).stop 2024-04-28T12:39:33.310 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.635+0000 7f33ea600700 1 -- 172.21.15.84:0/1473745491 shutdown_connections 2024-04-28T12:39:33.310 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.635+0000 7f33ea600700 1 --2- 172.21.15.84:0/1473745491 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33e407ee20 0x7f33e419f090 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:33.310 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.635+0000 7f33ea600700 1 -- 172.21.15.84:0/1473745491 >> 172.21.15.84:0/1473745491 conn(0x7f33e4104470 msgr2=0x7f33e41050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:33.310 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.635+0000 7f33ea600700 1 -- 172.21.15.84:0/1473745491 shutdown_connections 2024-04-28T12:39:33.310 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:32.636+0000 7f33ea600700 1 -- 172.21.15.84:0/1473745491 wait complete. 2024-04-28T12:39:33.310 INFO:teuthology.orchestra.run.smithi084.stdout:mgr not available, waiting (1/15)... 2024-04-28T12:39:34.917 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:34 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:34.516+0000 7feb1a348200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T12:39:34.917 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:34 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:34.740+0000 7feb1a348200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T12:39:35.323 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:34 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:34.952+0000 7feb1a348200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T12:39:35.323 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:35 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:35.327+0000 7feb1a348200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T12:39:35.616 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:35 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:35.453+0000 7feb1a348200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T12:39:35.866 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:35 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:35.711+0000 7feb1a348200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T12:39:36.167 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:35 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:35.967+0000 7feb1a348200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:39:36.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:36 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/548635543' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-28T12:39:36.701 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:36.702 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:39:36.702 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "ddee604a-055b-11ef-bc93-c7b262605968", 2024-04-28T12:39:36.702 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-04-28T12:39:36.702 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-28T12:39:36.702 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-28T12:39:36.702 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-28T12:39:36.702 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:36.702 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-28T12:39:36.702 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-28T12:39:36.702 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-04-28T12:39:36.702 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:39:36.702 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-28T12:39:36.703 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-28T12:39:36.711 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-28T12:39:36.711 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-28T12:39:36.712 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:36.713 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-04-28T12:39:22.249243+0000", 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.924+0000 7f9ae4e00700 1 Processor -- start 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.925+0000 7f9ae4e00700 1 -- start start 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.925+0000 7f9ae4e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 0x7f9ae007d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.925+0000 7f9ae4e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae007d7f0 con 0x7f9ae007ee00 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.925+0000 7f9adde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 0x7f9ae007d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.926+0000 7f9adde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 0x7f9ae007d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:44572/0 (socket says 172.21.15.84:44572) 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.926+0000 7f9adde00700 1 -- 172.21.15.84:0/1173340905 learned_addr learned my addr 172.21.15.84:0/1173340905 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:36.714 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.926+0000 7f9adde00700 1 -- 172.21.15.84:0/1173340905 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae007d970 con 0x7f9ae007ee00 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.926+0000 7f9adde00700 1 --2- 172.21.15.84:0/1173340905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 0x7f9ae007d2b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9ad00098a0 tx=0x7f9ad0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88caac8ec0e5248c server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.926+0000 7f9add400700 1 -- 172.21.15.84:0/1173340905 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ad0004160 con 0x7f9ae007ee00 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.926+0000 7f9add400700 1 -- 172.21.15.84:0/1173340905 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9ad0004300 con 0x7f9ae007ee00 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.927+0000 7f9add400700 1 -- 172.21.15.84:0/1173340905 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ad0011af0 con 0x7f9ae007ee00 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.927+0000 7f9ae4e00700 1 -- 172.21.15.84:0/1173340905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 msgr2=0x7f9ae007d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.927+0000 7f9ae4e00700 1 --2- 172.21.15.84:0/1173340905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 0x7f9ae007d2b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9ad00098a0 tx=0x7f9ad0009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.927+0000 7f9ae4e00700 1 -- 172.21.15.84:0/1173340905 shutdown_connections 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.927+0000 7f9ae4e00700 1 --2- 172.21.15.84:0/1173340905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 0x7f9ae007d2b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.927+0000 7f9ae4e00700 1 -- 172.21.15.84:0/1173340905 >> 172.21.15.84:0/1173340905 conn(0x7f9ae0104470 msgr2=0x7f9ae01068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.928+0000 7f9ae4e00700 1 -- 172.21.15.84:0/1173340905 shutdown_connections 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.928+0000 7f9ae4e00700 1 -- 172.21.15.84:0/1173340905 wait complete. 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.929+0000 7f9ae4e00700 1 Processor -- start 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.929+0000 7f9ae4e00700 1 -- start start 2024-04-28T12:39:36.715 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.929+0000 7f9ae4e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 0x7f9ae019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.929+0000 7f9ae4e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae010f4a0 con 0x7f9ae007ee00 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.929+0000 7f9adde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 0x7f9ae019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.929+0000 7f9adde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 0x7f9ae019f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:44586/0 (socket says 172.21.15.84:44586) 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.929+0000 7f9adde00700 1 -- 172.21.15.84:0/548635543 learned_addr learned my addr 172.21.15.84:0/548635543 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.930+0000 7f9adde00700 1 -- 172.21.15.84:0/548635543 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae019f5f0 con 0x7f9ae007ee00 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.930+0000 7f9adde00700 1 --2- 172.21.15.84:0/548635543 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 0x7f9ae019f0b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0011ac0 tx=0x7f9ad0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.930+0000 7f9ad6a00700 1 -- 172.21.15.84:0/548635543 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ad0003970 con 0x7f9ae007ee00 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.930+0000 7f9ad6a00700 1 -- 172.21.15.84:0/548635543 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9ad0003b10 con 0x7f9ae007ee00 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.930+0000 7f9ae4e00700 1 -- 172.21.15.84:0/548635543 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae019f880 con 0x7f9ae007ee00 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.930+0000 7f9ad6a00700 1 -- 172.21.15.84:0/548635543 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ad0024610 con 0x7f9ae007ee00 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.930+0000 7f9ae4e00700 1 -- 172.21.15.84:0/548635543 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae01a2570 con 0x7f9ae007ee00 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.931+0000 7f9ad6a00700 1 -- 172.21.15.84:0/548635543 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9ad0024bd0 con 0x7f9ae007ee00 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.931+0000 7f9ad6a00700 1 -- 172.21.15.84:0/548635543 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9ad0019700 con 0x7f9ae007ee00 2024-04-28T12:39:36.716 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.932+0000 7f9ae4e00700 1 -- 172.21.15.84:0/548635543 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ae0064a80 con 0x7f9ae007ee00 2024-04-28T12:39:36.717 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:35.934+0000 7f9ad6a00700 1 -- 172.21.15.84:0/548635543 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f9ad0043ec0 con 0x7f9ae007ee00 2024-04-28T12:39:36.717 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:36.014+0000 7f9ae4e00700 1 -- 172.21.15.84:0/548635543 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9ae01a2740 con 0x7f9ae007ee00 2024-04-28T12:39:36.717 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:36.015+0000 7f9ad6a00700 1 -- 172.21.15.84:0/548635543 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9ad0032030 con 0x7f9ae007ee00 2024-04-28T12:39:36.717 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:36.017+0000 7f9ae4e00700 1 -- 172.21.15.84:0/548635543 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 msgr2=0x7f9ae019f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:36.717 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:36.017+0000 7f9ae4e00700 1 --2- 172.21.15.84:0/548635543 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 0x7f9ae019f0b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0011ac0 tx=0x7f9ad0011ba0 comp rx=0 tx=0).stop 2024-04-28T12:39:36.717 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:36.017+0000 7f9ae4e00700 1 -- 172.21.15.84:0/548635543 shutdown_connections 2024-04-28T12:39:36.717 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:36.017+0000 7f9ae4e00700 1 --2- 172.21.15.84:0/548635543 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ae007ee00 0x7f9ae019f0b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:36.717 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:36.017+0000 7f9ae4e00700 1 -- 172.21.15.84:0/548635543 >> 172.21.15.84:0/548635543 conn(0x7f9ae0104470 msgr2=0x7f9ae01050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:36.717 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:36.017+0000 7f9ae4e00700 1 -- 172.21.15.84:0/548635543 shutdown_connections 2024-04-28T12:39:36.717 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:36.017+0000 7f9ae4e00700 1 -- 172.21.15.84:0/548635543 wait complete. 2024-04-28T12:39:36.717 INFO:teuthology.orchestra.run.smithi084.stdout:mgr not available, waiting (2/15)... 2024-04-28T12:39:37.167 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:36 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:36.789+0000 7feb1a348200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T12:39:37.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:37 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:37.335+0000 7feb1a348200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T12:39:37.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:37 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:37.460+0000 7feb1a348200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T12:39:37.975 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:37 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:37.718+0000 7feb1a348200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T12:39:38.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:38 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:38.353+0000 7feb1a348200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T12:39:38.668 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:38 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:38.474+0000 7feb1a348200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T12:39:39.048 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:38 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:38.795+0000 7feb1a348200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T12:39:39.048 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:38 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:38.973+0000 7feb1a348200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T12:39:39.299 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:39 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:39.106+0000 7feb1a348200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T12:39:39.299 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:39 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:39.272+0000 7feb1a348200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T12:39:39.917 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:39 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:39.682+0000 7feb1a348200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T12:39:39.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:39 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2391892746' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-28T12:39:40.289 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:40.289 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:39:40.289 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "ddee604a-055b-11ef-bc93-c7b262605968", 2024-04-28T12:39:40.289 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:40.290 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-28T12:39:40.291 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-28T12:39:40.291 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:40.291 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-28T12:39:40.291 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:40.291 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-28T12:39:40.291 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-28T12:39:40.291 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-28T12:39:40.300 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-28T12:39:40.300 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-28T12:39:40.300 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-28T12:39:40.300 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:40.300 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-28T12:39:40.300 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-28T12:39:40.300 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-28T12:39:40.300 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-28T12:39:40.300 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-28T12:39:40.300 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-28T12:39:40.301 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-04-28T12:39:22.249243+0000", 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.473+0000 7fd495600700 1 Processor -- start 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.473+0000 7fd495600700 1 -- start start 2024-04-28T12:39:40.302 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.474+0000 7fd495600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 0x7fd49010be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.474+0000 7fd495600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd49007cde0 con 0x7fd490109a80 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.474+0000 7fd486a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 0x7fd49010be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.474+0000 7fd486a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 0x7fd49010be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:34968/0 (socket says 172.21.15.84:34968) 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.474+0000 7fd486a00700 1 -- 172.21.15.84:0/2163005492 learned_addr learned my addr 172.21.15.84:0/2163005492 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.475+0000 7fd486a00700 1 -- 172.21.15.84:0/2163005492 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49010c3b0 con 0x7fd490109a80 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.475+0000 7fd486a00700 1 --2- 172.21.15.84:0/2163005492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 0x7fd49010be70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd4800098a0 tx=0x7fd480009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595af1fbde1f4919 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.476+0000 7fd48e000700 1 -- 172.21.15.84:0/2163005492 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd480004160 con 0x7fd490109a80 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.476+0000 7fd48e000700 1 -- 172.21.15.84:0/2163005492 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd480004300 con 0x7fd490109a80 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.476+0000 7fd48e000700 1 -- 172.21.15.84:0/2163005492 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd480011af0 con 0x7fd490109a80 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.476+0000 7fd495600700 1 -- 172.21.15.84:0/2163005492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 msgr2=0x7fd49010be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.476+0000 7fd495600700 1 --2- 172.21.15.84:0/2163005492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 0x7fd49010be70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd4800098a0 tx=0x7fd480009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.476+0000 7fd495600700 1 -- 172.21.15.84:0/2163005492 shutdown_connections 2024-04-28T12:39:40.303 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.476+0000 7fd495600700 1 --2- 172.21.15.84:0/2163005492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 0x7fd49010be70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.476+0000 7fd495600700 1 -- 172.21.15.84:0/2163005492 >> 172.21.15.84:0/2163005492 conn(0x7fd490103410 msgr2=0x7fd490105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.477+0000 7fd495600700 1 -- 172.21.15.84:0/2163005492 shutdown_connections 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.477+0000 7fd495600700 1 -- 172.21.15.84:0/2163005492 wait complete. 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.478+0000 7fd495600700 1 Processor -- start 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.478+0000 7fd495600700 1 -- start start 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.478+0000 7fd495600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 0x7fd49019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.478+0000 7fd495600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd49010d250 con 0x7fd490109a80 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.479+0000 7fd486a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 0x7fd49019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.479+0000 7fd486a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 0x7fd49019f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:34978/0 (socket says 172.21.15.84:34978) 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.479+0000 7fd486a00700 1 -- 172.21.15.84:0/2391892746 learned_addr learned my addr 172.21.15.84:0/2391892746 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.479+0000 7fd486a00700 1 -- 172.21.15.84:0/2391892746 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49019f590 con 0x7fd490109a80 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.479+0000 7fd486a00700 1 --2- 172.21.15.84:0/2391892746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 0x7fd49019f050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd480011ac0 tx=0x7fd480011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.480+0000 7fd487e00700 1 -- 172.21.15.84:0/2391892746 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd480003b20 con 0x7fd490109a80 2024-04-28T12:39:40.304 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.480+0000 7fd487e00700 1 -- 172.21.15.84:0/2391892746 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd48001a8c0 con 0x7fd490109a80 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.480+0000 7fd487e00700 1 -- 172.21.15.84:0/2391892746 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4800248d0 con 0x7fd490109a80 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.480+0000 7fd495600700 1 -- 172.21.15.84:0/2391892746 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd49019f820 con 0x7fd490109a80 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.480+0000 7fd495600700 1 -- 172.21.15.84:0/2391892746 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4901a2510 con 0x7fd490109a80 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.480+0000 7fd487e00700 1 -- 172.21.15.84:0/2391892746 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd480022020 con 0x7fd490109a80 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.480+0000 7fd487e00700 1 -- 172.21.15.84:0/2391892746 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd4800193c0 con 0x7fd490109a80 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.481+0000 7fd495600700 1 -- 172.21.15.84:0/2391892746 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd490064a80 con 0x7fd490109a80 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.484+0000 7fd487e00700 1 -- 172.21.15.84:0/2391892746 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fd48002ddf0 con 0x7fd490109a80 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.562+0000 7fd495600700 1 -- 172.21.15.84:0/2391892746 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd4901a2a50 con 0x7fd490109a80 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.563+0000 7fd487e00700 1 -- 172.21.15.84:0/2391892746 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7fd48001aa70 con 0x7fd490109a80 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.564+0000 7fd495600700 1 -- 172.21.15.84:0/2391892746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 msgr2=0x7fd49019f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.564+0000 7fd495600700 1 --2- 172.21.15.84:0/2391892746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 0x7fd49019f050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd480011ac0 tx=0x7fd480011ba0 comp rx=0 tx=0).stop 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.565+0000 7fd495600700 1 -- 172.21.15.84:0/2391892746 shutdown_connections 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.565+0000 7fd495600700 1 --2- 172.21.15.84:0/2391892746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd490109a80 0x7fd49019f050 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.565+0000 7fd495600700 1 -- 172.21.15.84:0/2391892746 >> 172.21.15.84:0/2391892746 conn(0x7fd490103410 msgr2=0x7fd490103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:40.305 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.565+0000 7fd495600700 1 -- 172.21.15.84:0/2391892746 shutdown_connections 2024-04-28T12:39:40.306 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:39.565+0000 7fd495600700 1 -- 172.21.15.84:0/2391892746 wait complete. 2024-04-28T12:39:40.306 INFO:teuthology.orchestra.run.smithi084.stdout:mgr not available, waiting (3/15)... 2024-04-28T12:39:40.615 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:40.471+0000 7feb1a348200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T12:39:40.917 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:40.752+0000 7feb1a348200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T12:39:40.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: Activating manager daemon x 2024-04-28T12:39:40.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: mgrmap e2: x(active, starting, since 0.0044069s) 2024-04-28T12:39:40.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: from='mgr.14100 172.21.15.84:0/1823806633' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-28T12:39:40.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: from='mgr.14100 172.21.15.84:0/1823806633' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-28T12:39:40.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: from='mgr.14100 172.21.15.84:0/1823806633' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-28T12:39:40.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: from='mgr.14100 172.21.15.84:0/1823806633' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:39:40.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: from='mgr.14100 172.21.15.84:0/1823806633' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-28T12:39:40.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: Manager daemon x is now available 2024-04-28T12:39:40.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: from='mgr.14100 172.21.15.84:0/1823806633' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-28T12:39:40.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: from='mgr.14100 172.21.15.84:0/1823806633' entity='mgr.x' 2024-04-28T12:39:40.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: from='mgr.14100 172.21.15.84:0/1823806633' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-28T12:39:40.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: from='mgr.14100 172.21.15.84:0/1823806633' entity='mgr.x' 2024-04-28T12:39:40.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:40 smithi084 ceph-mon[28081]: from='mgr.14100 172.21.15.84:0/1823806633' entity='mgr.x' 2024-04-28T12:39:43.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:42 smithi084 ceph-mon[28081]: mgrmap e3: x(active, since 1.01657s) 2024-04-28T12:39:43.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:43 smithi084 ceph-mon[28081]: mgrmap e4: x(active, since 2s) 2024-04-28T12:39:43.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:43 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/336322283' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-28T12:39:43.856 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:43.859 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:39:43.859 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "ddee604a-055b-11ef-bc93-c7b262605968", 2024-04-28T12:39:43.859 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-04-28T12:39:43.859 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-28T12:39:43.859 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-28T12:39:43.859 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-28T12:39:43.859 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:43.859 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-28T12:39:43.860 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-28T12:39:43.869 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-28T12:39:43.869 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-28T12:39:43.869 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-28T12:39:43.869 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-28T12:39:43.869 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:43.869 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-28T12:39:43.869 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:43.870 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-04-28T12:39:22.249243+0000", 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-04-28T12:39:43.871 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.893+0000 7f5b5bc00700 1 Processor -- start 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.893+0000 7f5b5bc00700 1 -- start start 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.894+0000 7f5b5bc00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 0x7f5b5410b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.894+0000 7f5b5bc00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5410bb30 con 0x7f5b5410b140 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.894+0000 7f5b59200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 0x7f5b5410b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.894+0000 7f5b59200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 0x7f5b5410b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35040/0 (socket says 172.21.15.84:35040) 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.894+0000 7f5b59200700 1 -- 172.21.15.84:0/1247600181 learned_addr learned my addr 172.21.15.84:0/1247600181 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.895+0000 7f5b59200700 1 -- 172.21.15.84:0/1247600181 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5410c2f0 con 0x7f5b5410b140 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.895+0000 7f5b59200700 1 --2- 172.21.15.84:0/1247600181 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 0x7f5b5410b560 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5b480098a0 tx=0x7f5b48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce313113c6588e20 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.895+0000 7f5b53400700 1 -- 172.21.15.84:0/1247600181 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b48004160 con 0x7f5b5410b140 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.895+0000 7f5b53400700 1 -- 172.21.15.84:0/1247600181 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5b48004300 con 0x7f5b5410b140 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.895+0000 7f5b53400700 1 -- 172.21.15.84:0/1247600181 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b48011af0 con 0x7f5b5410b140 2024-04-28T12:39:43.872 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.896+0000 7f5b5bc00700 1 -- 172.21.15.84:0/1247600181 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 msgr2=0x7f5b5410b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.896+0000 7f5b5bc00700 1 --2- 172.21.15.84:0/1247600181 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 0x7f5b5410b560 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5b480098a0 tx=0x7f5b48009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.896+0000 7f5b5bc00700 1 -- 172.21.15.84:0/1247600181 shutdown_connections 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.896+0000 7f5b5bc00700 1 --2- 172.21.15.84:0/1247600181 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 0x7f5b5410b560 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.896+0000 7f5b5bc00700 1 -- 172.21.15.84:0/1247600181 >> 172.21.15.84:0/1247600181 conn(0x7f5b5407e6b0 msgr2=0x7f5b5407eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.896+0000 7f5b5bc00700 1 -- 172.21.15.84:0/1247600181 shutdown_connections 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.896+0000 7f5b5bc00700 1 -- 172.21.15.84:0/1247600181 wait complete. 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.897+0000 7f5b5bc00700 1 Processor -- start 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.898+0000 7f5b5bc00700 1 -- start start 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.898+0000 7f5b5bc00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 0x7f5b5419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.898+0000 7f5b5bc00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5410f6a0 con 0x7f5b5410b140 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.898+0000 7f5b59200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 0x7f5b5419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.898+0000 7f5b59200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 0x7f5b5419f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35050/0 (socket says 172.21.15.84:35050) 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.898+0000 7f5b59200700 1 -- 172.21.15.84:0/336322283 learned_addr learned my addr 172.21.15.84:0/336322283 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.899+0000 7f5b59200700 1 -- 172.21.15.84:0/336322283 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5419f5e0 con 0x7f5b5410b140 2024-04-28T12:39:43.873 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.899+0000 7f5b59200700 1 --2- 172.21.15.84:0/336322283 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 0x7f5b5419f0a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009fb0 tx=0x7f5b48011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:43.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.899+0000 7f5b51600700 1 -- 172.21.15.84:0/336322283 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b480036a0 con 0x7f5b5410b140 2024-04-28T12:39:43.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.899+0000 7f5b51600700 1 -- 172.21.15.84:0/336322283 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5b4801a430 con 0x7f5b5410b140 2024-04-28T12:39:43.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.899+0000 7f5b5bc00700 1 -- 172.21.15.84:0/336322283 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b5419f870 con 0x7f5b5410b140 2024-04-28T12:39:43.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.899+0000 7f5b51600700 1 -- 172.21.15.84:0/336322283 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b48019550 con 0x7f5b5410b140 2024-04-28T12:39:43.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.899+0000 7f5b5bc00700 1 -- 172.21.15.84:0/336322283 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b541a2560 con 0x7f5b5410b140 2024-04-28T12:39:43.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.900+0000 7f5b51600700 1 -- 172.21.15.84:0/336322283 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f5b48003840 con 0x7f5b5410b140 2024-04-28T12:39:43.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.900+0000 7f5b5bc00700 1 -- 172.21.15.84:0/336322283 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b54064a80 con 0x7f5b5410b140 2024-04-28T12:39:43.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.901+0000 7f5b51600700 1 --2- 172.21.15.84:0/336322283 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f5b3c039560 0x7f5b3c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:43.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.901+0000 7f5b51600700 1 -- 172.21.15.84:0/336322283 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5b4804d850 con 0x7f5b5410b140 2024-04-28T12:39:43.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.901+0000 7f5b53e00700 1 --2- 172.21.15.84:0/336322283 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f5b3c039560 0x7f5b3c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:43.874 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.902+0000 7f5b53e00700 1 --2- 172.21.15.84:0/336322283 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f5b3c039560 0x7f5b3c03ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5b40006fd0 tx=0x7f5b40006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:42.907+0000 7f5b51600700 1 -- 172.21.15.84:0/336322283 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5b4804d270 con 0x7f5b5410b140 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:43.218+0000 7f5b5bc00700 1 -- 172.21.15.84:0/336322283 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f5b541a2730 con 0x7f5b5410b140 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:43.219+0000 7f5b51600700 1 -- 172.21.15.84:0/336322283 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f5b541a2730 con 0x7f5b5410b140 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:43.221+0000 7f5b5bc00700 1 -- 172.21.15.84:0/336322283 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f5b3c039560 msgr2=0x7f5b3c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:43.221+0000 7f5b5bc00700 1 --2- 172.21.15.84:0/336322283 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f5b3c039560 0x7f5b3c03ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5b40006fd0 tx=0x7f5b40006e40 comp rx=0 tx=0).stop 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:43.221+0000 7f5b5bc00700 1 -- 172.21.15.84:0/336322283 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 msgr2=0x7f5b5419f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:43.221+0000 7f5b5bc00700 1 --2- 172.21.15.84:0/336322283 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 0x7f5b5419f0a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009fb0 tx=0x7f5b48011940 comp rx=0 tx=0).stop 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:43.221+0000 7f5b5bc00700 1 -- 172.21.15.84:0/336322283 shutdown_connections 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:43.221+0000 7f5b5bc00700 1 --2- 172.21.15.84:0/336322283 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f5b3c039560 0x7f5b3c03ba20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:43.221+0000 7f5b5bc00700 1 --2- 172.21.15.84:0/336322283 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b5410b140 0x7f5b5419f0a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:43.221+0000 7f5b5bc00700 1 -- 172.21.15.84:0/336322283 >> 172.21.15.84:0/336322283 conn(0x7f5b5407e6b0 msgr2=0x7f5b54108510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:43.221+0000 7f5b5bc00700 1 -- 172.21.15.84:0/336322283 shutdown_connections 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:43.221+0000 7f5b5bc00700 1 -- 172.21.15.84:0/336322283 wait complete. 2024-04-28T12:39:43.875 INFO:teuthology.orchestra.run.smithi084.stdout:mgr is available 2024-04-28T12:39:45.108 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:44 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1818974922' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-28T12:39:45.415 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:45.415 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [global] 2024-04-28T12:39:45.415 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout fsid = ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:39:45.415 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-28T12:39:45.415 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-28T12:39:45.415 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-28T12:39:45.415 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-28T12:39:45.415 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-28T12:39:45.416 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:45.416 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-28T12:39:45.416 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-28T12:39:45.416 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-04-28T12:39:45.416 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [osd] 2024-04-28T12:39:45.422 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-04-28T12:39:45.422 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-28T12:39:45.422 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-28T12:39:45.422 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-28T12:39:45.422 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-28T12:39:45.422 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-28T12:39:45.422 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.464+0000 7ff28ee00700 1 Processor -- start 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.464+0000 7ff28ee00700 1 -- start start 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.465+0000 7ff28ee00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 0x7ff2881120b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.465+0000 7ff28ee00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff288112680 con 0x7ff28810fcc0 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.465+0000 7ff287e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 0x7ff2881120b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.465+0000 7ff287e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 0x7ff2881120b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35066/0 (socket says 172.21.15.84:35066) 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.465+0000 7ff287e00700 1 -- 172.21.15.84:0/1835375033 learned_addr learned my addr 172.21.15.84:0/1835375033 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.466+0000 7ff287e00700 1 -- 172.21.15.84:0/1835375033 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff288112800 con 0x7ff28810fcc0 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.466+0000 7ff287e00700 1 --2- 172.21.15.84:0/1835375033 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 0x7ff2881120b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff2780098a0 tx=0x7ff278009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d91f2703b5b34c server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.467+0000 7ff287400700 1 -- 172.21.15.84:0/1835375033 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff278004160 con 0x7ff28810fcc0 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.467+0000 7ff287400700 1 -- 172.21.15.84:0/1835375033 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff278004300 con 0x7ff28810fcc0 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.467+0000 7ff287400700 1 -- 172.21.15.84:0/1835375033 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff278011af0 con 0x7ff28810fcc0 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.467+0000 7ff28ee00700 1 -- 172.21.15.84:0/1835375033 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 msgr2=0x7ff2881120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.468+0000 7ff28ee00700 1 --2- 172.21.15.84:0/1835375033 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 0x7ff2881120b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff2780098a0 tx=0x7ff278009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:45.423 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.468+0000 7ff28ee00700 1 -- 172.21.15.84:0/1835375033 shutdown_connections 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.468+0000 7ff28ee00700 1 --2- 172.21.15.84:0/1835375033 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 0x7ff2881120b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.468+0000 7ff28ee00700 1 -- 172.21.15.84:0/1835375033 >> 172.21.15.84:0/1835375033 conn(0x7ff28807e540 msgr2=0x7ff288080960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.468+0000 7ff28ee00700 1 -- 172.21.15.84:0/1835375033 shutdown_connections 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.468+0000 7ff28ee00700 1 -- 172.21.15.84:0/1835375033 wait complete. 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.469+0000 7ff28ee00700 1 Processor -- start 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.469+0000 7ff28ee00700 1 -- start start 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.469+0000 7ff28ee00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 0x7ff2881a33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.469+0000 7ff28ee00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff288113880 con 0x7ff28810fcc0 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.470+0000 7ff287e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 0x7ff2881a33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.470+0000 7ff287e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 0x7ff2881a33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35080/0 (socket says 172.21.15.84:35080) 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.470+0000 7ff287e00700 1 -- 172.21.15.84:0/1818974922 learned_addr learned my addr 172.21.15.84:0/1818974922 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.470+0000 7ff287e00700 1 -- 172.21.15.84:0/1818974922 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2881a3920 con 0x7ff28810fcc0 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.470+0000 7ff287e00700 1 --2- 172.21.15.84:0/1818974922 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 0x7ff2881a33e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff278009fb0 tx=0x7ff278011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.470+0000 7ff285600700 1 -- 172.21.15.84:0/1818974922 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2780036a0 con 0x7ff28810fcc0 2024-04-28T12:39:45.424 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.471+0000 7ff285600700 1 -- 172.21.15.84:0/1818974922 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff27801a430 con 0x7ff28810fcc0 2024-04-28T12:39:45.425 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.471+0000 7ff28ee00700 1 -- 172.21.15.84:0/1818974922 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2881a3b50 con 0x7ff28810fcc0 2024-04-28T12:39:45.425 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.471+0000 7ff285600700 1 -- 172.21.15.84:0/1818974922 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff278019550 con 0x7ff28810fcc0 2024-04-28T12:39:45.425 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.471+0000 7ff28ee00700 1 -- 172.21.15.84:0/1818974922 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2881a68a0 con 0x7ff28810fcc0 2024-04-28T12:39:45.425 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.472+0000 7ff285600700 1 -- 172.21.15.84:0/1818974922 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7ff278003840 con 0x7ff28810fcc0 2024-04-28T12:39:45.425 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.472+0000 7ff285600700 1 --2- 172.21.15.84:0/1818974922 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7ff26c039560 0x7ff26c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:45.425 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.472+0000 7ff28ee00700 1 -- 172.21.15.84:0/1818974922 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff288064a80 con 0x7ff28810fcc0 2024-04-28T12:39:45.425 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.472+0000 7ff285600700 1 -- 172.21.15.84:0/1818974922 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff27804d830 con 0x7ff28810fcc0 2024-04-28T12:39:45.425 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.472+0000 7ff27fe00700 1 --2- 172.21.15.84:0/1818974922 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7ff26c039560 0x7ff26c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:45.425 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.473+0000 7ff27fe00700 1 --2- 172.21.15.84:0/1818974922 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7ff26c039560 0x7ff26c03ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff270006fd0 tx=0x7ff270006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:45.425 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.478+0000 7ff285600700 1 -- 172.21.15.84:0/1818974922 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff278023e00 con 0x7ff28810fcc0 2024-04-28T12:39:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.699+0000 7ff28ee00700 1 -- 172.21.15.84:0/1818974922 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7ff2881a6a70 con 0x7ff28810fcc0 2024-04-28T12:39:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.701+0000 7ff285600700 1 -- 172.21.15.84:0/1818974922 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+434 (secure 0 0 0) 0x7ff278003b30 con 0x7ff28810fcc0 2024-04-28T12:39:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.704+0000 7ff28ee00700 1 -- 172.21.15.84:0/1818974922 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7ff26c039560 msgr2=0x7ff26c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.704+0000 7ff28ee00700 1 --2- 172.21.15.84:0/1818974922 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7ff26c039560 0x7ff26c03ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff270006fd0 tx=0x7ff270006e40 comp rx=0 tx=0).stop 2024-04-28T12:39:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.704+0000 7ff28ee00700 1 -- 172.21.15.84:0/1818974922 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 msgr2=0x7ff2881a33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.704+0000 7ff28ee00700 1 --2- 172.21.15.84:0/1818974922 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 0x7ff2881a33e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff278009fb0 tx=0x7ff278011940 comp rx=0 tx=0).stop 2024-04-28T12:39:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.705+0000 7ff28ee00700 1 -- 172.21.15.84:0/1818974922 shutdown_connections 2024-04-28T12:39:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.705+0000 7ff28ee00700 1 --2- 172.21.15.84:0/1818974922 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7ff26c039560 0x7ff26c03ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.705+0000 7ff28ee00700 1 --2- 172.21.15.84:0/1818974922 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28810fcc0 0x7ff2881a33e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.705+0000 7ff28ee00700 1 -- 172.21.15.84:0/1818974922 >> 172.21.15.84:0/1818974922 conn(0x7ff28807e540 msgr2=0x7ff28807f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.705+0000 7ff28ee00700 1 -- 172.21.15.84:0/1818974922 shutdown_connections 2024-04-28T12:39:45.427 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:44.705+0000 7ff28ee00700 1 -- 172.21.15.84:0/1818974922 wait complete. 2024-04-28T12:39:45.427 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling cephadm module... 2024-04-28T12:39:46.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:46 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3599675073' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-28T12:39:46.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:46 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: ignoring --setuser ceph since I am not root 2024-04-28T12:39:46.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:46 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: ignoring --setgroup ceph since I am not root 2024-04-28T12:39:47.112 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:46 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:46.749+0000 7fc46f742200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T12:39:47.112 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:46 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:46.986+0000 7fc46f742200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:39:47.234 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.128+0000 7fb254000700 1 Processor -- start 2024-04-28T12:39:47.234 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.128+0000 7fb254000700 1 -- start start 2024-04-28T12:39:47.234 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.129+0000 7fb254000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 0x7fb24c10b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.129+0000 7fb254000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c10bb30 con 0x7fb24c10b140 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.129+0000 7fb251600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 0x7fb24c10b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.129+0000 7fb251600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 0x7fb24c10b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35094/0 (socket says 172.21.15.84:35094) 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.129+0000 7fb251600700 1 -- 172.21.15.84:0/554567253 learned_addr learned my addr 172.21.15.84:0/554567253 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.130+0000 7fb251600700 1 -- 172.21.15.84:0/554567253 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24c10c2f0 con 0x7fb24c10b140 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.130+0000 7fb251600700 1 --2- 172.21.15.84:0/554567253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 0x7fb24c10b560 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb2400098a0 tx=0x7fb240009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c27bdbed995b7322 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.131+0000 7fb24be00700 1 -- 172.21.15.84:0/554567253 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb240004160 con 0x7fb24c10b140 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.131+0000 7fb24be00700 1 -- 172.21.15.84:0/554567253 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb240004300 con 0x7fb24c10b140 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.131+0000 7fb24be00700 1 -- 172.21.15.84:0/554567253 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb240011af0 con 0x7fb24c10b140 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.131+0000 7fb254000700 1 -- 172.21.15.84:0/554567253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 msgr2=0x7fb24c10b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.131+0000 7fb254000700 1 --2- 172.21.15.84:0/554567253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 0x7fb24c10b560 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb2400098a0 tx=0x7fb240009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.131+0000 7fb254000700 1 -- 172.21.15.84:0/554567253 shutdown_connections 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.131+0000 7fb254000700 1 --2- 172.21.15.84:0/554567253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 0x7fb24c10b560 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.131+0000 7fb254000700 1 -- 172.21.15.84:0/554567253 >> 172.21.15.84:0/554567253 conn(0x7fb24c07e4d0 msgr2=0x7fb24c07e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.132+0000 7fb254000700 1 -- 172.21.15.84:0/554567253 shutdown_connections 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.132+0000 7fb254000700 1 -- 172.21.15.84:0/554567253 wait complete. 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.133+0000 7fb254000700 1 Processor -- start 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.133+0000 7fb254000700 1 -- start start 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.133+0000 7fb254000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 0x7fb24c19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.133+0000 7fb254000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c10f6a0 con 0x7fb24c10b140 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.133+0000 7fb251600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 0x7fb24c19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.133+0000 7fb251600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 0x7fb24c19f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35102/0 (socket says 172.21.15.84:35102) 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.133+0000 7fb251600700 1 -- 172.21.15.84:0/3599675073 learned_addr learned my addr 172.21.15.84:0/3599675073 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.134+0000 7fb251600700 1 -- 172.21.15.84:0/3599675073 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24c19f5e0 con 0x7fb24c10b140 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.134+0000 7fb251600700 1 --2- 172.21.15.84:0/3599675073 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 0x7fb24c19f0a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb240000c00 tx=0x7fb240011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.134+0000 7fb24a000700 1 -- 172.21.15.84:0/3599675073 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb240003b20 con 0x7fb24c10b140 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.134+0000 7fb24a000700 1 -- 172.21.15.84:0/3599675073 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb24001a8c0 con 0x7fb24c10b140 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.134+0000 7fb254000700 1 -- 172.21.15.84:0/3599675073 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb24c19f870 con 0x7fb24c10b140 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.134+0000 7fb24a000700 1 -- 172.21.15.84:0/3599675073 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb240024940 con 0x7fb24c10b140 2024-04-28T12:39:47.236 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.135+0000 7fb254000700 1 -- 172.21.15.84:0/3599675073 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb24c1a2560 con 0x7fb24c10b140 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.136+0000 7fb24a000700 1 -- 172.21.15.84:0/3599675073 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fb240022020 con 0x7fb24c10b140 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.136+0000 7fb254000700 1 -- 172.21.15.84:0/3599675073 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb24c064a80 con 0x7fb24c10b140 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.136+0000 7fb24a000700 1 --2- 172.21.15.84:0/3599675073 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7fb2340395b0 0x7fb23403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.136+0000 7fb24a000700 1 -- 172.21.15.84:0/3599675073 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb24004d450 con 0x7fb24c10b140 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.136+0000 7fb250c00700 1 --2- 172.21.15.84:0/3599675073 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7fb2340395b0 0x7fb23403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.137+0000 7fb250c00700 1 --2- 172.21.15.84:0/3599675073 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7fb2340395b0 0x7fb23403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb238006fd0 tx=0x7fb238006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.141+0000 7fb24a000700 1 -- 172.21.15.84:0/3599675073 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb24002c3a0 con 0x7fb24c10b140 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.420+0000 7fb254000700 1 -- 172.21.15.84:0/3599675073 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fb24c1a2730 con 0x7fb24c10b140 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.473+0000 7fb24a000700 1 -- 172.21.15.84:0/3599675073 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fb24c1a2730 con 0x7fb24c10b140 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.477+0000 7fb24a000700 1 -- 172.21.15.84:0/3599675073 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fb24001aa70 con 0x7fb24c10b140 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.478+0000 7fb254000700 1 -- 172.21.15.84:0/3599675073 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7fb2340395b0 msgr2=0x7fb23403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.478+0000 7fb254000700 1 --2- 172.21.15.84:0/3599675073 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7fb2340395b0 0x7fb23403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb238006fd0 tx=0x7fb238006e40 comp rx=0 tx=0).stop 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.478+0000 7fb254000700 1 -- 172.21.15.84:0/3599675073 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 msgr2=0x7fb24c19f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:47.237 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.478+0000 7fb254000700 1 --2- 172.21.15.84:0/3599675073 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 0x7fb24c19f0a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb240000c00 tx=0x7fb240011b70 comp rx=0 tx=0).stop 2024-04-28T12:39:47.238 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.478+0000 7fb254000700 1 -- 172.21.15.84:0/3599675073 shutdown_connections 2024-04-28T12:39:47.238 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.478+0000 7fb254000700 1 --2- 172.21.15.84:0/3599675073 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7fb2340395b0 0x7fb23403ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:47.238 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.478+0000 7fb254000700 1 --2- 172.21.15.84:0/3599675073 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb24c10b140 0x7fb24c19f0a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:47.238 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.478+0000 7fb254000700 1 -- 172.21.15.84:0/3599675073 >> 172.21.15.84:0/3599675073 conn(0x7fb24c07e4d0 msgr2=0x7fb24c1084f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:47.238 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.479+0000 7fb254000700 1 -- 172.21.15.84:0/3599675073 shutdown_connections 2024-04-28T12:39:47.238 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:46.479+0000 7fb254000700 1 -- 172.21.15.84:0/3599675073 wait complete. 2024-04-28T12:39:47.374 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:47 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:47.115+0000 7fc46f742200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T12:39:47.636 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:47 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3599675073' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-28T12:39:47.636 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:47 smithi084 ceph-mon[28081]: mgrmap e5: x(active, since 5s) 2024-04-28T12:39:47.636 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:47 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:47.522+0000 7fc46f742200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T12:39:47.917 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:47 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:47.639+0000 7fc46f742200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T12:39:48.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:48 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4102017648' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-28T12:39:48.782 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:39:48.783 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-28T12:39:48.783 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-28T12:39:48.786 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-28T12:39:48.786 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.812+0000 7f88c3a00700 1 Processor -- start 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.813+0000 7f88c3a00700 1 -- start start 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.813+0000 7f88c3a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 0x7f88bc07d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.813+0000 7f88c3a00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc07d900 con 0x7f88bc07ee80 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.813+0000 7f88c1000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 0x7f88bc07d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.814+0000 7f88c1000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 0x7f88bc07d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35124/0 (socket says 172.21.15.84:35124) 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.814+0000 7f88c1000700 1 -- 172.21.15.84:0/218473429 learned_addr learned my addr 172.21.15.84:0/218473429 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.814+0000 7f88c1000700 1 -- 172.21.15.84:0/218473429 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88bc07da80 con 0x7f88bc07ee80 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.814+0000 7f88c1000700 1 --2- 172.21.15.84:0/218473429 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 0x7f88bc07d330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f88b00098a0 tx=0x7f88b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aede32fe409e4d server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.814+0000 7f88bb400700 1 -- 172.21.15.84:0/218473429 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88b0004160 con 0x7f88bc07ee80 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.815+0000 7f88bb400700 1 -- 172.21.15.84:0/218473429 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f88b0004300 con 0x7f88bc07ee80 2024-04-28T12:39:48.787 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.815+0000 7f88bb400700 1 -- 172.21.15.84:0/218473429 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88b0011af0 con 0x7f88bc07ee80 2024-04-28T12:39:48.788 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.815+0000 7f88c3a00700 1 -- 172.21.15.84:0/218473429 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 msgr2=0x7f88bc07d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:48.788 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.815+0000 7f88c3a00700 1 --2- 172.21.15.84:0/218473429 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 0x7f88bc07d330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f88b00098a0 tx=0x7f88b0009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:48.788 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.815+0000 7f88c3a00700 1 -- 172.21.15.84:0/218473429 shutdown_connections 2024-04-28T12:39:48.788 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.815+0000 7f88c3a00700 1 --2- 172.21.15.84:0/218473429 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 0x7f88bc07d330 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:48.788 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.815+0000 7f88c3a00700 1 -- 172.21.15.84:0/218473429 >> 172.21.15.84:0/218473429 conn(0x7f88bc104470 msgr2=0x7f88bc1068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:48.788 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.815+0000 7f88c3a00700 1 -- 172.21.15.84:0/218473429 shutdown_connections 2024-04-28T12:39:48.788 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.816+0000 7f88c3a00700 1 -- 172.21.15.84:0/218473429 wait complete. 2024-04-28T12:39:48.788 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.816+0000 7f88c3a00700 1 Processor -- start 2024-04-28T12:39:48.788 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.817+0000 7f88c3a00700 1 -- start start 2024-04-28T12:39:48.788 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.817+0000 7f88c3a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 0x7f88bc1a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:48.792 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.817+0000 7f88c3a00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc10b070 con 0x7f88bc07ee80 2024-04-28T12:39:48.792 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.817+0000 7f88c1000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 0x7f88bc1a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:48.792 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.817+0000 7f88c1000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 0x7f88bc1a34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35134/0 (socket says 172.21.15.84:35134) 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.817+0000 7f88c1000700 1 -- 172.21.15.84:0/4102017648 learned_addr learned my addr 172.21.15.84:0/4102017648 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.818+0000 7f88c1000700 1 -- 172.21.15.84:0/4102017648 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88bc1a39f0 con 0x7f88bc07ee80 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.818+0000 7f88c1000700 1 --2- 172.21.15.84:0/4102017648 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 0x7f88bc1a34b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f88b0009fb0 tx=0x7f88b0011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.818+0000 7f88b9600700 1 -- 172.21.15.84:0/4102017648 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88b00036a0 con 0x7f88bc07ee80 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.818+0000 7f88b9600700 1 -- 172.21.15.84:0/4102017648 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f88b001a430 con 0x7f88bc07ee80 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.818+0000 7f88b9600700 1 -- 172.21.15.84:0/4102017648 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88b0019550 con 0x7f88bc07ee80 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.818+0000 7f88c3a00700 1 -- 172.21.15.84:0/4102017648 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88bc1a3c80 con 0x7f88bc07ee80 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.818+0000 7f88c3a00700 1 -- 172.21.15.84:0/4102017648 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88bc1a6970 con 0x7f88bc07ee80 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.819+0000 7f88b9600700 1 -- 172.21.15.84:0/4102017648 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f88b0003840 con 0x7f88bc07ee80 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.819+0000 7f88c3a00700 1 -- 172.21.15.84:0/4102017648 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88bc064a80 con 0x7f88bc07ee80 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.820+0000 7f88b9600700 1 --2- 172.21.15.84:0/4102017648 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f88a40395b0 0x7f88a403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.820+0000 7f88b9600700 1 -- 172.21.15.84:0/4102017648 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f88b004d870 con 0x7f88bc07ee80 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.820+0000 7f88bbe00700 1 -- 172.21.15.84:0/4102017648 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f88a40395b0 msgr2=0x7f88a403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4148140420 2024-04-28T12:39:48.793 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.820+0000 7f88bbe00700 1 --2- 172.21.15.84:0/4102017648 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f88a40395b0 0x7f88a403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:47.825+0000 7f88b9600700 1 -- 172.21.15.84:0/4102017648 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f88b00172b0 con 0x7f88bc07ee80 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.021+0000 7f88bbe00700 1 -- 172.21.15.84:0/4102017648 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f88a40395b0 msgr2=0x7f88a403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4148140420 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.021+0000 7f88bbe00700 1 --2- 172.21.15.84:0/4102017648 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f88a40395b0 0x7f88a403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.100+0000 7f88c3a00700 1 -- 172.21.15.84:0/4102017648 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f88bc1a6b40 con 0x7f88bc07ee80 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.100+0000 7f88b9600700 1 -- 172.21.15.84:0/4102017648 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f88bc1a6b40 con 0x7f88bc07ee80 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.103+0000 7f88c3a00700 1 -- 172.21.15.84:0/4102017648 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f88a40395b0 msgr2=0x7f88a403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.103+0000 7f88c3a00700 1 --2- 172.21.15.84:0/4102017648 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f88a40395b0 0x7f88a403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.103+0000 7f88c3a00700 1 -- 172.21.15.84:0/4102017648 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 msgr2=0x7f88bc1a34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.103+0000 7f88c3a00700 1 --2- 172.21.15.84:0/4102017648 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 0x7f88bc1a34b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f88b0009fb0 tx=0x7f88b0011940 comp rx=0 tx=0).stop 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.103+0000 7f88c3a00700 1 -- 172.21.15.84:0/4102017648 shutdown_connections 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.103+0000 7f88c3a00700 1 --2- 172.21.15.84:0/4102017648 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f88a40395b0 0x7f88a403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.103+0000 7f88c3a00700 1 --2- 172.21.15.84:0/4102017648 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f88bc07ee80 0x7f88bc1a34b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.103+0000 7f88c3a00700 1 -- 172.21.15.84:0/4102017648 >> 172.21.15.84:0/4102017648 conn(0x7f88bc104470 msgr2=0x7f88bc1068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:48.794 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.104+0000 7f88c3a00700 1 -- 172.21.15.84:0/4102017648 shutdown_connections 2024-04-28T12:39:48.795 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:48.104+0000 7f88c3a00700 1 -- 172.21.15.84:0/4102017648 wait complete. 2024-04-28T12:39:48.795 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for the mgr to restart... 2024-04-28T12:39:48.795 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr epoch 5... 2024-04-28T12:39:49.348 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:49 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:49.205+0000 7fc46f742200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T12:39:49.646 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:49 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:49.438+0000 7fc46f742200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T12:39:49.646 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:49 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:49.649+0000 7fc46f742200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T12:39:50.391 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:50 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:50.019+0000 7fc46f742200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T12:39:50.392 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:50 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:50.146+0000 7fc46f742200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T12:39:50.649 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:50 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:50.394+0000 7fc46f742200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T12:39:50.649 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:50 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:50.652+0000 7fc46f742200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:39:51.917 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:51 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:51.454+0000 7fc46f742200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T12:39:52.361 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:51 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:51.995+0000 7fc46f742200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T12:39:52.362 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:52 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:52.123+0000 7fc46f742200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T12:39:52.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:52 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:52.388+0000 7fc46f742200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T12:39:53.417 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:53 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:53.029+0000 7fc46f742200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T12:39:53.417 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:53 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:53.150+0000 7fc46f742200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T12:39:53.762 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:53 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:53.467+0000 7fc46f742200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T12:39:53.762 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:53 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:53.639+0000 7fc46f742200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T12:39:54.167 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:53 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:53.765+0000 7fc46f742200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T12:39:54.167 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:53 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:53.916+0000 7fc46f742200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T12:39:54.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:54 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:54.284+0000 7fc46f742200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T12:39:55.304 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:55.026+0000 7fc46f742200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T12:39:55.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:39:55.308+0000 7fc46f742200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T12:39:55.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-mon[28081]: Active manager daemon x restarted 2024-04-28T12:39:55.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-mon[28081]: Activating manager daemon x 2024-04-28T12:39:55.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-mon[28081]: osdmap e2: 0 total, 0 up, 0 in 2024-04-28T12:39:55.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-mon[28081]: mgrmap e6: x(active, starting, since 0.00485154s) 2024-04-28T12:39:55.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:39:55.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-28T12:39:55.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-28T12:39:55.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-28T12:39:55.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-28T12:39:55.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-mon[28081]: Manager daemon x is now available 2024-04-28T12:39:55.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:39:55.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:55 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:39:56.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:56 smithi084 ceph-mon[28081]: Found migration_current of "None". Setting to last migration. 2024-04-28T12:39:56.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:56 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:39:56.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:56 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:39:56.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:56 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-28T12:39:56.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:56 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-28T12:39:56.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:56 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:39:56.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:56 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:39:56.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:56 smithi084 ceph-mon[28081]: mgrmap e7: x(active, since 1.01741s) 2024-04-28T12:39:57.079 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:39:57.079 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-28T12:39:57.079 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-28T12:39:57.079 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:39:57.079 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.351+0000 7f0691e00700 1 Processor -- start 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.351+0000 7f0691e00700 1 -- start start 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.352+0000 7f0691e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 0x7f068c10be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.352+0000 7f0691e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f068c07cde0 con 0x7f068c109a80 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.352+0000 7f068aa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 0x7f068c10be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.352+0000 7f068aa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 0x7f068c10be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42348/0 (socket says 172.21.15.84:42348) 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.352+0000 7f068aa00700 1 -- 172.21.15.84:0/1403238916 learned_addr learned my addr 172.21.15.84:0/1403238916 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.353+0000 7f068aa00700 1 -- 172.21.15.84:0/1403238916 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c10c3b0 con 0x7f068c109a80 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.353+0000 7f068aa00700 1 --2- 172.21.15.84:0/1403238916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 0x7f068c10be70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f067c0098a0 tx=0x7f067c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b580332d1161e92 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.353+0000 7f0689600700 1 -- 172.21.15.84:0/1403238916 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f067c004160 con 0x7f068c109a80 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.353+0000 7f0689600700 1 -- 172.21.15.84:0/1403238916 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f067c004300 con 0x7f068c109a80 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.353+0000 7f0689600700 1 -- 172.21.15.84:0/1403238916 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f067c011af0 con 0x7f068c109a80 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.354+0000 7f0691e00700 1 -- 172.21.15.84:0/1403238916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 msgr2=0x7f068c10be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.354+0000 7f0691e00700 1 --2- 172.21.15.84:0/1403238916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 0x7f068c10be70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f067c0098a0 tx=0x7f067c009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.354+0000 7f0691e00700 1 -- 172.21.15.84:0/1403238916 shutdown_connections 2024-04-28T12:39:57.080 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.354+0000 7f0691e00700 1 --2- 172.21.15.84:0/1403238916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 0x7f068c10be70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.354+0000 7f0691e00700 1 -- 172.21.15.84:0/1403238916 >> 172.21.15.84:0/1403238916 conn(0x7f068c103410 msgr2=0x7f068c105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.354+0000 7f0691e00700 1 -- 172.21.15.84:0/1403238916 shutdown_connections 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.354+0000 7f0691e00700 1 -- 172.21.15.84:0/1403238916 wait complete. 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.355+0000 7f0691e00700 1 Processor -- start 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.356+0000 7f0691e00700 1 -- start start 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.356+0000 7f0691e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 0x7f068c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.356+0000 7f0691e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f068c10d250 con 0x7f068c109a80 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.356+0000 7f068aa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 0x7f068c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.356+0000 7f068aa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 0x7f068c19f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42362/0 (socket says 172.21.15.84:42362) 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.356+0000 7f068aa00700 1 -- 172.21.15.84:0/3666092252 learned_addr learned my addr 172.21.15.84:0/3666092252 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.357+0000 7f068aa00700 1 -- 172.21.15.84:0/3666092252 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c19f570 con 0x7f068c109a80 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.357+0000 7f068aa00700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 0x7f068c19f030 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f067c011ac0 tx=0x7f067c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.357+0000 7f0690a00700 1 -- 172.21.15.84:0/3666092252 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f067c003b20 con 0x7f068c109a80 2024-04-28T12:39:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.357+0000 7f0690a00700 1 -- 172.21.15.84:0/3666092252 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f067c01a8c0 con 0x7f068c109a80 2024-04-28T12:39:57.082 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.357+0000 7f0690a00700 1 -- 172.21.15.84:0/3666092252 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f067c0248d0 con 0x7f068c109a80 2024-04-28T12:39:57.082 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.357+0000 7f0691e00700 1 -- 172.21.15.84:0/3666092252 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068c19f800 con 0x7f068c109a80 2024-04-28T12:39:57.082 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.357+0000 7f0691e00700 1 -- 172.21.15.84:0/3666092252 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f068c1a24f0 con 0x7f068c109a80 2024-04-28T12:39:57.082 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.359+0000 7f0690a00700 1 -- 172.21.15.84:0/3666092252 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f067c022020 con 0x7f068c109a80 2024-04-28T12:39:57.082 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.359+0000 7f0690a00700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 0x7f067003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:57.082 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.360+0000 7f0690a00700 1 -- 172.21.15.84:0/3666092252 --> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f067003c1d0 con 0x7f0670039600 2024-04-28T12:39:57.082 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.360+0000 7f0690a00700 1 -- 172.21.15.84:0/3666092252 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f067c04d570 con 0x7f068c109a80 2024-04-28T12:39:57.082 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.360+0000 7f068a000700 1 -- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 msgr2=0x7f067003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4148140420 2024-04-28T12:39:57.082 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.360+0000 7f068a000700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 0x7f067003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:39:57.082 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.561+0000 7f068a000700 1 -- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 msgr2=0x7f067003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4148140420 2024-04-28T12:39:57.082 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.561+0000 7f068a000700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 0x7f067003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-28T12:39:57.082 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.962+0000 7f068a000700 1 -- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 msgr2=0x7f067003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4148140420 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:49.962+0000 7f068a000700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 0x7f067003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:50.763+0000 7f068a000700 1 -- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 msgr2=0x7f067003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4148140420 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:50.763+0000 7f068a000700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 0x7f067003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:52.365+0000 7f068a000700 1 -- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 msgr2=0x7f067003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4148140420 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:52.365+0000 7f068a000700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 0x7f067003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:55.315+0000 7f0690a00700 1 -- 172.21.15.84:0/3666092252 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f067c02d3f0 con 0x7f068c109a80 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:55.315+0000 7f0690a00700 1 -- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 msgr2=0x7f067003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:55.315+0000 7f0690a00700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 0x7f067003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.323+0000 7f0690a00700 1 -- 172.21.15.84:0/3666092252 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f067c04ee20 con 0x7f068c109a80 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.323+0000 7f0690a00700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f067003cef0 0x7f067003f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.323+0000 7f0690a00700 1 -- 172.21.15.84:0/3666092252 --> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f067c00f4c0 con 0x7f067003cef0 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.326+0000 7f068a000700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f067003cef0 0x7f067003f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:57.083 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.329+0000 7f068a000700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f067003cef0 0x7f067003f2e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0678003a10 tx=0x7f06780092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.332+0000 7f0690a00700 1 -- 172.21.15.84:0/3666092252 <== mgr.14120 v2:172.21.15.84:6800/4213070051 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f067c00f4c0 con 0x7f067003cef0 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.347+0000 7f0691e00700 1 -- 172.21.15.84:0/3666092252 --> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f068c06a590 con 0x7f067003cef0 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.347+0000 7f0690a00700 1 -- 172.21.15.84:0/3666092252 <== mgr.14120 v2:172.21.15.84:6800/4213070051 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f068c06a590 con 0x7f067003cef0 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.348+0000 7f0691e00700 1 -- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f067003cef0 msgr2=0x7f067003f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.348+0000 7f0691e00700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f067003cef0 0x7f067003f2e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0678003a10 tx=0x7f06780092b0 comp rx=0 tx=0).stop 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.348+0000 7f0691e00700 1 -- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 msgr2=0x7f068c19f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.348+0000 7f0691e00700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 0x7f068c19f030 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f067c011ac0 tx=0x7f067c011ba0 comp rx=0 tx=0).stop 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.348+0000 7f0691e00700 1 -- 172.21.15.84:0/3666092252 shutdown_connections 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.348+0000 7f0691e00700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f067003cef0 0x7f067003f2e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.348+0000 7f0691e00700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f068c109a80 0x7f068c19f030 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.348+0000 7f0691e00700 1 --2- 172.21.15.84:0/3666092252 >> [v2:172.21.15.84:6800/4148140420,v1:172.21.15.84:6801/4148140420] conn(0x7f0670039600 0x7f067003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.348+0000 7f0691e00700 1 -- 172.21.15.84:0/3666092252 >> 172.21.15.84:0/3666092252 conn(0x7f068c103410 msgr2=0x7f068c103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.349+0000 7f0691e00700 1 -- 172.21.15.84:0/3666092252 shutdown_connections 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:56.349+0000 7f0691e00700 1 -- 172.21.15.84:0/3666092252 wait complete. 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:mgr epoch 5 is available 2024-04-28T12:39:57.084 INFO:teuthology.orchestra.run.smithi084.stdout:Setting orchestrator backend to cephadm... 2024-04-28T12:39:57.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:57 smithi084 ceph-mon[28081]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-28T12:39:57.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:57 smithi084 ceph-mon[28081]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-28T12:39:57.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:57 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:39:58.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:58 smithi084 ceph-mon[28081]: [28/Apr/2024:12:39:56] ENGINE Bus STARTING 2024-04-28T12:39:58.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:58 smithi084 ceph-mon[28081]: [28/Apr/2024:12:39:57] ENGINE Serving on https://172.21.15.84:7150 2024-04-28T12:39:58.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:58 smithi084 ceph-mon[28081]: [28/Apr/2024:12:39:57] ENGINE Serving on http://172.21.15.84:8765 2024-04-28T12:39:58.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:58 smithi084 ceph-mon[28081]: [28/Apr/2024:12:39:57] ENGINE Bus STARTED 2024-04-28T12:39:58.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:58 smithi084 ceph-mon[28081]: mgrmap e8: x(active, since 2s) 2024-04-28T12:39:58.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:58 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:39:58.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:58 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:39:58.597 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.690+0000 7f77e3a00700 1 Processor -- start 2024-04-28T12:39:58.598 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.690+0000 7f77e3a00700 1 -- start start 2024-04-28T12:39:58.598 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.691+0000 7f77e3a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 0x7f77dc07d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:58.598 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.691+0000 7f77e3a00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f77dc07d810 con 0x7f77dc07ee20 2024-04-28T12:39:58.598 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.691+0000 7f77e1000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 0x7f77dc07d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:58.599 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.691+0000 7f77e1000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 0x7f77dc07d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42432/0 (socket says 172.21.15.84:42432) 2024-04-28T12:39:58.599 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.691+0000 7f77e1000700 1 -- 172.21.15.84:0/1632356361 learned_addr learned my addr 172.21.15.84:0/1632356361 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:58.599 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.692+0000 7f77e1000700 1 -- 172.21.15.84:0/1632356361 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77dc07d990 con 0x7f77dc07ee20 2024-04-28T12:39:58.599 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.692+0000 7f77e1000700 1 --2- 172.21.15.84:0/1632356361 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 0x7f77dc07d2d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f77d00098a0 tx=0x7f77d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc49147112996fc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:58.600 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.692+0000 7f77db400700 1 -- 172.21.15.84:0/1632356361 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77d0004160 con 0x7f77dc07ee20 2024-04-28T12:39:58.600 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.692+0000 7f77db400700 1 -- 172.21.15.84:0/1632356361 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f77d0004300 con 0x7f77dc07ee20 2024-04-28T12:39:58.600 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.692+0000 7f77db400700 1 -- 172.21.15.84:0/1632356361 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77d0011af0 con 0x7f77dc07ee20 2024-04-28T12:39:58.600 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.693+0000 7f77e3a00700 1 -- 172.21.15.84:0/1632356361 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 msgr2=0x7f77dc07d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:58.601 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.693+0000 7f77e3a00700 1 --2- 172.21.15.84:0/1632356361 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 0x7f77dc07d2d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f77d00098a0 tx=0x7f77d0009bb0 comp rx=0 tx=0).stop 2024-04-28T12:39:58.601 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.693+0000 7f77e3a00700 1 -- 172.21.15.84:0/1632356361 shutdown_connections 2024-04-28T12:39:58.601 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.693+0000 7f77e3a00700 1 --2- 172.21.15.84:0/1632356361 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 0x7f77dc07d2d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:58.602 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.693+0000 7f77e3a00700 1 -- 172.21.15.84:0/1632356361 >> 172.21.15.84:0/1632356361 conn(0x7f77dc104470 msgr2=0x7f77dc1068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:58.602 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.693+0000 7f77e3a00700 1 -- 172.21.15.84:0/1632356361 shutdown_connections 2024-04-28T12:39:58.602 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.693+0000 7f77e3a00700 1 -- 172.21.15.84:0/1632356361 wait complete. 2024-04-28T12:39:58.602 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.694+0000 7f77e3a00700 1 Processor -- start 2024-04-28T12:39:58.602 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.694+0000 7f77e3a00700 1 -- start start 2024-04-28T12:39:58.603 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.695+0000 7f77e3a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 0x7f77dc19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:58.603 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.695+0000 7f77e3a00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f77dc10f480 con 0x7f77dc07ee20 2024-04-28T12:39:58.603 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.695+0000 7f77e1000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 0x7f77dc19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:58.603 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.695+0000 7f77e1000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 0x7f77dc19f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42446/0 (socket says 172.21.15.84:42446) 2024-04-28T12:39:58.603 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.695+0000 7f77e1000700 1 -- 172.21.15.84:0/1195901522 learned_addr learned my addr 172.21.15.84:0/1195901522 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:39:58.604 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.695+0000 7f77e1000700 1 -- 172.21.15.84:0/1195901522 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77dc19f5d0 con 0x7f77dc07ee20 2024-04-28T12:39:58.604 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.695+0000 7f77e1000700 1 --2- 172.21.15.84:0/1195901522 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 0x7f77dc19f090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f77d0011ac0 tx=0x7f77d0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:58.604 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.696+0000 7f77d9600700 1 -- 172.21.15.84:0/1195901522 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77d0003970 con 0x7f77dc07ee20 2024-04-28T12:39:58.604 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.696+0000 7f77d9600700 1 -- 172.21.15.84:0/1195901522 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f77d0003b10 con 0x7f77dc07ee20 2024-04-28T12:39:58.604 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.696+0000 7f77e3a00700 1 -- 172.21.15.84:0/1195901522 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77dc19f860 con 0x7f77dc07ee20 2024-04-28T12:39:58.605 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.696+0000 7f77d9600700 1 -- 172.21.15.84:0/1195901522 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77d0024610 con 0x7f77dc07ee20 2024-04-28T12:39:58.605 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.696+0000 7f77e3a00700 1 -- 172.21.15.84:0/1195901522 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77dc1a2550 con 0x7f77dc07ee20 2024-04-28T12:39:58.605 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.697+0000 7f77d9600700 1 -- 172.21.15.84:0/1195901522 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f77d001a9e0 con 0x7f77dc07ee20 2024-04-28T12:39:58.605 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.697+0000 7f77e3a00700 1 -- 172.21.15.84:0/1195901522 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77dc064a80 con 0x7f77dc07ee20 2024-04-28T12:39:58.605 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.697+0000 7f77d9600700 1 --2- 172.21.15.84:0/1195901522 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f77c4039600 0x7f77c403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:39:58.605 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.698+0000 7f77d9600700 1 -- 172.21.15.84:0/1195901522 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f77d004d9a0 con 0x7f77dc07ee20 2024-04-28T12:39:58.606 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.698+0000 7f77dbe00700 1 --2- 172.21.15.84:0/1195901522 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f77c4039600 0x7f77c403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:39:58.607 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.698+0000 7f77dbe00700 1 --2- 172.21.15.84:0/1195901522 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f77c4039600 0x7f77c403bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f77c8006fd0 tx=0x7f77c8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:39:58.607 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.703+0000 7f77d9600700 1 -- 172.21.15.84:0/1195901522 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f77d000b3e0 con 0x7f77dc07ee20 2024-04-28T12:39:58.607 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.945+0000 7f77e3a00700 1 -- 172.21.15.84:0/1195901522 --> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f77dc1057b0 con 0x7f77c4039600 2024-04-28T12:39:58.607 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.955+0000 7f77d9600700 1 -- 172.21.15.84:0/1195901522 <== mgr.14120 v2:172.21.15.84:6800/4213070051 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f77dc1057b0 con 0x7f77c4039600 2024-04-28T12:39:58.607 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.959+0000 7f77e3a00700 1 -- 172.21.15.84:0/1195901522 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f77c4039600 msgr2=0x7f77c403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:58.607 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.959+0000 7f77e3a00700 1 --2- 172.21.15.84:0/1195901522 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f77c4039600 0x7f77c403bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f77c8006fd0 tx=0x7f77c8006e40 comp rx=0 tx=0).stop 2024-04-28T12:39:58.608 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.959+0000 7f77e3a00700 1 -- 172.21.15.84:0/1195901522 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 msgr2=0x7f77dc19f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:39:58.608 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.959+0000 7f77e3a00700 1 --2- 172.21.15.84:0/1195901522 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 0x7f77dc19f090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f77d0011ac0 tx=0x7f77d0011ba0 comp rx=0 tx=0).stop 2024-04-28T12:39:58.608 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.959+0000 7f77e3a00700 1 -- 172.21.15.84:0/1195901522 shutdown_connections 2024-04-28T12:39:58.608 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.959+0000 7f77e3a00700 1 --2- 172.21.15.84:0/1195901522 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f77c4039600 0x7f77c403bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:58.608 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.959+0000 7f77e3a00700 1 --2- 172.21.15.84:0/1195901522 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f77dc07ee20 0x7f77dc19f090 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:39:58.608 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.959+0000 7f77e3a00700 1 -- 172.21.15.84:0/1195901522 >> 172.21.15.84:0/1195901522 conn(0x7f77dc104470 msgr2=0x7f77dc1050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:39:58.608 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.959+0000 7f77e3a00700 1 -- 172.21.15.84:0/1195901522 shutdown_connections 2024-04-28T12:39:58.609 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:57.960+0000 7f77e3a00700 1 -- 172.21.15.84:0/1195901522 wait complete. 2024-04-28T12:39:59.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:39:59 smithi084 ceph-mon[28081]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:40:00.116 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-28T12:40:00.116 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.197+0000 7f7a59e00700 1 Processor -- start 2024-04-28T12:40:00.117 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.198+0000 7f7a59e00700 1 -- start start 2024-04-28T12:40:00.117 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.198+0000 7f7a59e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 0x7f7a5410b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:00.117 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.198+0000 7f7a59e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5410bb30 con 0x7f7a5410b140 2024-04-28T12:40:00.117 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.199+0000 7f7a52a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 0x7f7a5410b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:00.117 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.199+0000 7f7a52a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 0x7f7a5410b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41194/0 (socket says 172.21.15.84:41194) 2024-04-28T12:40:00.117 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.199+0000 7f7a52a00700 1 -- 172.21.15.84:0/2314771719 learned_addr learned my addr 172.21.15.84:0/2314771719 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:00.117 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.199+0000 7f7a52a00700 1 -- 172.21.15.84:0/2314771719 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a5410c340 con 0x7f7a5410b140 2024-04-28T12:40:00.117 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.199+0000 7f7a52a00700 1 --2- 172.21.15.84:0/2314771719 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 0x7f7a5410b560 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7a440098a0 tx=0x7f7a44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d142b881a37701 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:00.117 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.200+0000 7f7a52000700 1 -- 172.21.15.84:0/2314771719 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a44004160 con 0x7f7a5410b140 2024-04-28T12:40:00.117 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.200+0000 7f7a52000700 1 -- 172.21.15.84:0/2314771719 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7a44004300 con 0x7f7a5410b140 2024-04-28T12:40:00.117 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.200+0000 7f7a52000700 1 -- 172.21.15.84:0/2314771719 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a44011af0 con 0x7f7a5410b140 2024-04-28T12:40:00.117 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.201+0000 7f7a59e00700 1 -- 172.21.15.84:0/2314771719 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 msgr2=0x7f7a5410b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.201+0000 7f7a59e00700 1 --2- 172.21.15.84:0/2314771719 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 0x7f7a5410b560 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7a440098a0 tx=0x7f7a44009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.201+0000 7f7a59e00700 1 -- 172.21.15.84:0/2314771719 shutdown_connections 2024-04-28T12:40:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.201+0000 7f7a59e00700 1 --2- 172.21.15.84:0/2314771719 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 0x7f7a5410b560 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.201+0000 7f7a59e00700 1 -- 172.21.15.84:0/2314771719 >> 172.21.15.84:0/2314771719 conn(0x7f7a5407e750 msgr2=0x7f7a5407eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.201+0000 7f7a59e00700 1 -- 172.21.15.84:0/2314771719 shutdown_connections 2024-04-28T12:40:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.201+0000 7f7a59e00700 1 -- 172.21.15.84:0/2314771719 wait complete. 2024-04-28T12:40:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.202+0000 7f7a59e00700 1 Processor -- start 2024-04-28T12:40:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.202+0000 7f7a59e00700 1 -- start start 2024-04-28T12:40:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.202+0000 7f7a59e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 0x7f7a5419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.202+0000 7f7a59e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5410f6f0 con 0x7f7a5410b140 2024-04-28T12:40:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.203+0000 7f7a52a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 0x7f7a5419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.203+0000 7f7a52a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 0x7f7a5419f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41202/0 (socket says 172.21.15.84:41202) 2024-04-28T12:40:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.203+0000 7f7a52a00700 1 -- 172.21.15.84:0/20001633 learned_addr learned my addr 172.21.15.84:0/20001633 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.203+0000 7f7a52a00700 1 -- 172.21.15.84:0/20001633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a5419f630 con 0x7f7a5410b140 2024-04-28T12:40:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.203+0000 7f7a52a00700 1 --2- 172.21.15.84:0/20001633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 0x7f7a5419f0f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7a440117b0 tx=0x7f7a44011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.203+0000 7f7a58a00700 1 -- 172.21.15.84:0/20001633 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a44003af0 con 0x7f7a5410b140 2024-04-28T12:40:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.204+0000 7f7a58a00700 1 -- 172.21.15.84:0/20001633 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7a44020070 con 0x7f7a5410b140 2024-04-28T12:40:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.204+0000 7f7a59e00700 1 -- 172.21.15.84:0/20001633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a5419f8c0 con 0x7f7a5410b140 2024-04-28T12:40:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.204+0000 7f7a58a00700 1 -- 172.21.15.84:0/20001633 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a44019630 con 0x7f7a5410b140 2024-04-28T12:40:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.204+0000 7f7a59e00700 1 -- 172.21.15.84:0/20001633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a541a25b0 con 0x7f7a5410b140 2024-04-28T12:40:00.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.205+0000 7f7a59e00700 1 -- 172.21.15.84:0/20001633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a5406a590 con 0x7f7a5410b140 2024-04-28T12:40:00.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.205+0000 7f7a58a00700 1 -- 172.21.15.84:0/20001633 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f7a4401a9c0 con 0x7f7a5410b140 2024-04-28T12:40:00.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.205+0000 7f7a58a00700 1 --2- 172.21.15.84:0/20001633 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f7a380395b0 0x7f7a3803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:00.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.205+0000 7f7a58a00700 1 -- 172.21.15.84:0/20001633 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7a4404dc00 con 0x7f7a5410b140 2024-04-28T12:40:00.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.206+0000 7f7a4be00700 1 --2- 172.21.15.84:0/20001633 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f7a380395b0 0x7f7a3803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:00.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.206+0000 7f7a4be00700 1 --2- 172.21.15.84:0/20001633 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f7a380395b0 0x7f7a3803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c006fd0 tx=0x7f7a3c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:00.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.211+0000 7f7a58a00700 1 -- 172.21.15.84:0/20001633 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7a4400e260 con 0x7f7a5410b140 2024-04-28T12:40:00.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.442+0000 7f7a59e00700 1 -- 172.21.15.84:0/20001633 --> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f7a54108be0 con 0x7f7a380395b0 2024-04-28T12:40:00.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.444+0000 7f7a58a00700 1 -- 172.21.15.84:0/20001633 <== mgr.14120 v2:172.21.15.84:6800/4213070051 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f7a54108be0 con 0x7f7a380395b0 2024-04-28T12:40:00.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.446+0000 7f7a59e00700 1 -- 172.21.15.84:0/20001633 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f7a380395b0 msgr2=0x7f7a3803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:00.127 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.446+0000 7f7a59e00700 1 --2- 172.21.15.84:0/20001633 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f7a380395b0 0x7f7a3803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c006fd0 tx=0x7f7a3c006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:00.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.446+0000 7f7a59e00700 1 -- 172.21.15.84:0/20001633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 msgr2=0x7f7a5419f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:00.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.446+0000 7f7a59e00700 1 --2- 172.21.15.84:0/20001633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 0x7f7a5419f0f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7a440117b0 tx=0x7f7a44011b70 comp rx=0 tx=0).stop 2024-04-28T12:40:00.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.446+0000 7f7a59e00700 1 -- 172.21.15.84:0/20001633 shutdown_connections 2024-04-28T12:40:00.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.446+0000 7f7a59e00700 1 --2- 172.21.15.84:0/20001633 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f7a380395b0 0x7f7a3803ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:00.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.446+0000 7f7a59e00700 1 --2- 172.21.15.84:0/20001633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7a5410b140 0x7f7a5419f0f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:00.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.447+0000 7f7a59e00700 1 -- 172.21.15.84:0/20001633 >> 172.21.15.84:0/20001633 conn(0x7f7a5407e750 msgr2=0x7f7a541084d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:00.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.447+0000 7f7a59e00700 1 -- 172.21.15.84:0/20001633 shutdown_connections 2024-04-28T12:40:00.128 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:39:59.447+0000 7f7a59e00700 1 -- 172.21.15.84:0/20001633 wait complete. 2024-04-28T12:40:00.128 INFO:teuthology.orchestra.run.smithi084.stdout:Generating ssh key... 2024-04-28T12:40:00.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:00 smithi084 ceph-mon[28081]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:40:01.419 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: Generating public/private rsa key pair. 2024-04-28T12:40:01.419 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: Your identification has been saved in /tmp/tmpr2g0zp7y/key. 2024-04-28T12:40:01.419 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: Your public key has been saved in /tmp/tmpr2g0zp7y/key.pub. 2024-04-28T12:40:01.419 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: The key fingerprint is: 2024-04-28T12:40:01.419 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: SHA256:D2kniI7Ak4W4P1dDgPYvuIwK2uWN88p0Ssd9+L1/U+0 ceph-ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:40:01.420 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: The key's randomart image is: 2024-04-28T12:40:01.420 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: +---[RSA 3072]----+ 2024-04-28T12:40:01.420 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: | .. | 2024-04-28T12:40:01.420 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: |. + . | 2024-04-28T12:40:01.420 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: |.o o . | 2024-04-28T12:40:01.420 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: |..o .o . . | 2024-04-28T12:40:01.420 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: |o+ ...+ S . .| 2024-04-28T12:40:01.420 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: | ooo.o.+ * o| 2024-04-28T12:40:01.420 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: |.o+.*.+ o o ..| 2024-04-28T12:40:01.420 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: |+.oO.* o . .E| 2024-04-28T12:40:01.420 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: |+ . *+o . oo...| 2024-04-28T12:40:01.420 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:01 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: +----[SHA256]-----+ 2024-04-28T12:40:01.821 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.775+0000 7fbc75e00700 1 Processor -- start 2024-04-28T12:40:01.821 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.775+0000 7fbc75e00700 1 -- start start 2024-04-28T12:40:01.821 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.775+0000 7fbc75e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 0x7fbc701120b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:01.821 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.775+0000 7fbc75e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbc70112680 con 0x7fbc7010fcc0 2024-04-28T12:40:01.822 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.776+0000 7fbc6ea00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 0x7fbc701120b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:01.822 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.776+0000 7fbc6ea00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 0x7fbc701120b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41214/0 (socket says 172.21.15.84:41214) 2024-04-28T12:40:01.826 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.776+0000 7fbc6ea00700 1 -- 172.21.15.84:0/982869893 learned_addr learned my addr 172.21.15.84:0/982869893 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:01.826 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.776+0000 7fbc6ea00700 1 -- 172.21.15.84:0/982869893 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc70112800 con 0x7fbc7010fcc0 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.777+0000 7fbc6ea00700 1 --2- 172.21.15.84:0/982869893 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 0x7fbc701120b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc600098a0 tx=0x7fbc60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33bff516301fba2 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.777+0000 7fbc6d600700 1 -- 172.21.15.84:0/982869893 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc60004160 con 0x7fbc7010fcc0 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.777+0000 7fbc6d600700 1 -- 172.21.15.84:0/982869893 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbc60004300 con 0x7fbc7010fcc0 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.777+0000 7fbc6d600700 1 -- 172.21.15.84:0/982869893 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc60011af0 con 0x7fbc7010fcc0 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.778+0000 7fbc75e00700 1 -- 172.21.15.84:0/982869893 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 msgr2=0x7fbc701120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.778+0000 7fbc75e00700 1 --2- 172.21.15.84:0/982869893 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 0x7fbc701120b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc600098a0 tx=0x7fbc60009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.778+0000 7fbc75e00700 1 -- 172.21.15.84:0/982869893 shutdown_connections 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.778+0000 7fbc75e00700 1 --2- 172.21.15.84:0/982869893 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 0x7fbc701120b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.778+0000 7fbc75e00700 1 -- 172.21.15.84:0/982869893 >> 172.21.15.84:0/982869893 conn(0x7fbc7007e540 msgr2=0x7fbc70080960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.778+0000 7fbc75e00700 1 -- 172.21.15.84:0/982869893 shutdown_connections 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.778+0000 7fbc75e00700 1 -- 172.21.15.84:0/982869893 wait complete. 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.779+0000 7fbc75e00700 1 Processor -- start 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.780+0000 7fbc75e00700 1 -- start start 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.780+0000 7fbc75e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 0x7fbc701a33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:01.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.780+0000 7fbc75e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbc70113880 con 0x7fbc7010fcc0 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.780+0000 7fbc6ea00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 0x7fbc701a33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.780+0000 7fbc6ea00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 0x7fbc701a33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41224/0 (socket says 172.21.15.84:41224) 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.780+0000 7fbc6ea00700 1 -- 172.21.15.84:0/1427759218 learned_addr learned my addr 172.21.15.84:0/1427759218 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.780+0000 7fbc6ea00700 1 -- 172.21.15.84:0/1427759218 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc701a3920 con 0x7fbc7010fcc0 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.781+0000 7fbc6ea00700 1 --2- 172.21.15.84:0/1427759218 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 0x7fbc701a33e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009fb0 tx=0x7fbc60011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.781+0000 7fbc74a00700 1 -- 172.21.15.84:0/1427759218 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc600036a0 con 0x7fbc7010fcc0 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.781+0000 7fbc74a00700 1 -- 172.21.15.84:0/1427759218 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbc6001a430 con 0x7fbc7010fcc0 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.781+0000 7fbc75e00700 1 -- 172.21.15.84:0/1427759218 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc701a3b50 con 0x7fbc7010fcc0 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.781+0000 7fbc74a00700 1 -- 172.21.15.84:0/1427759218 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc60019550 con 0x7fbc7010fcc0 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.781+0000 7fbc75e00700 1 -- 172.21.15.84:0/1427759218 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc701a68a0 con 0x7fbc7010fcc0 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.782+0000 7fbc74a00700 1 -- 172.21.15.84:0/1427759218 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fbc6001a8e0 con 0x7fbc7010fcc0 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.782+0000 7fbc75e00700 1 -- 172.21.15.84:0/1427759218 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc70064a80 con 0x7fbc7010fcc0 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.783+0000 7fbc74a00700 1 --2- 172.21.15.84:0/1427759218 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbc540395b0 0x7fbc5403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:01.828 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.783+0000 7fbc74a00700 1 -- 172.21.15.84:0/1427759218 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbc6004d990 con 0x7fbc7010fcc0 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.783+0000 7fbc6e000700 1 --2- 172.21.15.84:0/1427759218 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbc540395b0 0x7fbc5403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.784+0000 7fbc6e000700 1 --2- 172.21.15.84:0/1427759218 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbc540395b0 0x7fbc5403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c006fd0 tx=0x7fbc5c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:00.789+0000 7fbc74a00700 1 -- 172.21.15.84:0/1427759218 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbc6000b3e0 con 0x7fbc7010fcc0 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:01.016+0000 7fbc75e00700 1 -- 172.21.15.84:0/1427759218 --> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fbc7007f780 con 0x7fbc540395b0 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:01.130+0000 7fbc74a00700 1 -- 172.21.15.84:0/1427759218 <== mgr.14120 v2:172.21.15.84:6800/4213070051 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbc7007f780 con 0x7fbc540395b0 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:01.134+0000 7fbc75e00700 1 -- 172.21.15.84:0/1427759218 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbc540395b0 msgr2=0x7fbc5403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:01.134+0000 7fbc75e00700 1 --2- 172.21.15.84:0/1427759218 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbc540395b0 0x7fbc5403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c006fd0 tx=0x7fbc5c006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:01.134+0000 7fbc75e00700 1 -- 172.21.15.84:0/1427759218 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 msgr2=0x7fbc701a33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:01.134+0000 7fbc75e00700 1 --2- 172.21.15.84:0/1427759218 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 0x7fbc701a33e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009fb0 tx=0x7fbc60011940 comp rx=0 tx=0).stop 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:01.134+0000 7fbc75e00700 1 -- 172.21.15.84:0/1427759218 shutdown_connections 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:01.134+0000 7fbc75e00700 1 --2- 172.21.15.84:0/1427759218 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbc540395b0 0x7fbc5403ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:01.134+0000 7fbc75e00700 1 --2- 172.21.15.84:0/1427759218 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc7010fcc0 0x7fbc701a33e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:01.134+0000 7fbc75e00700 1 -- 172.21.15.84:0/1427759218 >> 172.21.15.84:0/1427759218 conn(0x7fbc7007e540 msgr2=0x7fbc7007f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:01.134+0000 7fbc75e00700 1 -- 172.21.15.84:0/1427759218 shutdown_connections 2024-04-28T12:40:01.829 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:01.134+0000 7fbc75e00700 1 -- 172.21.15.84:0/1427759218 wait complete. 2024-04-28T12:40:02.397 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:02 smithi084 ceph-mon[28081]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:40:02.397 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:02 smithi084 ceph-mon[28081]: Generating ssh key... 2024-04-28T12:40:02.397 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:02 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:40:02.398 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:02 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:40:03.404 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAjknDkjQ0gVmdgzc9aJj0bv5gInLLZBiuM3xYG/OSQQzQWfNUrO8P3q9aoLvCprZPGnvJoebwrm3Z/gBNVVeZWKReRB60sk2jRPfu0LhoQtIURNrxaQrMNa0Si+d5jHGqT4PSeWgzYeejapVna1FoRCJKr7rfrzlRFCPL7QbXBfckSFJwpD4lCmNKvdvprjEHoe23Jbcmbt4Luyl9buRQXLW8uKe5ZnPf2MuvK4PImPxSLKtNy/aLTcWwi+wv+xGVEeIczB9FxsVNAqUGyiyHejjXu0NS7bVMgNuHyyL0ay1S1rZ5iuaYPUjWXVqIyJhrkitZVWbqoGDFIJ9HJPDZsO26LfHt+KJJhXNlYBCaR+TuSeq4MsaIChnirG5zfpUD0CdD2IbwNO2C0CbP/lhx5ZPv1rfumbmnOokY0zgZXEPvlRetDGbC3Z+41HxtUobUGF7Y9dI+mi1aF6BK1BxGoH2ZHOHoLpVTf7y4nN4/R/b+CqBaXCNA6snsnt+aMac= ceph-ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:40:03.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.454+0000 7f7e13400700 1 Processor -- start 2024-04-28T12:40:03.410 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.454+0000 7f7e13400700 1 -- start start 2024-04-28T12:40:03.410 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.455+0000 7f7e13400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 0x7f7e0c07d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:03.410 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.455+0000 7f7e13400700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0c07d810 con 0x7f7e0c07ee20 2024-04-28T12:40:03.410 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.455+0000 7f7e10a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 0x7f7e0c07d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:03.410 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.455+0000 7f7e10a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 0x7f7e0c07d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41228/0 (socket says 172.21.15.84:41228) 2024-04-28T12:40:03.410 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.455+0000 7f7e10a00700 1 -- 172.21.15.84:0/1715119003 learned_addr learned my addr 172.21.15.84:0/1715119003 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:03.410 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.456+0000 7f7e10a00700 1 -- 172.21.15.84:0/1715119003 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0c07d990 con 0x7f7e0c07ee20 2024-04-28T12:40:03.410 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.456+0000 7f7e10a00700 1 --2- 172.21.15.84:0/1715119003 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 0x7f7e0c07d2d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7e000098a0 tx=0x7f7e00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa958b9d55aeb032 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:03.410 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.456+0000 7f7e0b400700 1 -- 172.21.15.84:0/1715119003 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e00004160 con 0x7f7e0c07ee20 2024-04-28T12:40:03.410 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.457+0000 7f7e0b400700 1 -- 172.21.15.84:0/1715119003 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7e00004300 con 0x7f7e0c07ee20 2024-04-28T12:40:03.410 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.457+0000 7f7e0b400700 1 -- 172.21.15.84:0/1715119003 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e00011af0 con 0x7f7e0c07ee20 2024-04-28T12:40:03.410 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.457+0000 7f7e13400700 1 -- 172.21.15.84:0/1715119003 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 msgr2=0x7f7e0c07d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.457+0000 7f7e13400700 1 --2- 172.21.15.84:0/1715119003 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 0x7f7e0c07d2d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7e000098a0 tx=0x7f7e00009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.457+0000 7f7e13400700 1 -- 172.21.15.84:0/1715119003 shutdown_connections 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.457+0000 7f7e13400700 1 --2- 172.21.15.84:0/1715119003 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 0x7f7e0c07d2d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.457+0000 7f7e13400700 1 -- 172.21.15.84:0/1715119003 >> 172.21.15.84:0/1715119003 conn(0x7f7e0c104470 msgr2=0x7f7e0c1068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.458+0000 7f7e13400700 1 -- 172.21.15.84:0/1715119003 shutdown_connections 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.458+0000 7f7e13400700 1 -- 172.21.15.84:0/1715119003 wait complete. 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.459+0000 7f7e13400700 1 Processor -- start 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.459+0000 7f7e13400700 1 -- start start 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.459+0000 7f7e13400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 0x7f7e0c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.459+0000 7f7e13400700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0c10f480 con 0x7f7e0c07ee20 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.459+0000 7f7e10a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 0x7f7e0c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.460+0000 7f7e10a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 0x7f7e0c19f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41232/0 (socket says 172.21.15.84:41232) 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.460+0000 7f7e10a00700 1 -- 172.21.15.84:0/2237183980 learned_addr learned my addr 172.21.15.84:0/2237183980 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:03.411 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.460+0000 7f7e10a00700 1 -- 172.21.15.84:0/2237183980 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0c19f5d0 con 0x7f7e0c07ee20 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.460+0000 7f7e10a00700 1 --2- 172.21.15.84:0/2237183980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 0x7f7e0c19f090 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7e00011ac0 tx=0x7f7e00011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.460+0000 7f7e09600700 1 -- 172.21.15.84:0/2237183980 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e00003970 con 0x7f7e0c07ee20 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.461+0000 7f7e09600700 1 -- 172.21.15.84:0/2237183980 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7e00003b10 con 0x7f7e0c07ee20 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.461+0000 7f7e13400700 1 -- 172.21.15.84:0/2237183980 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e0c19f860 con 0x7f7e0c07ee20 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.461+0000 7f7e09600700 1 -- 172.21.15.84:0/2237183980 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e00024610 con 0x7f7e0c07ee20 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.461+0000 7f7e13400700 1 -- 172.21.15.84:0/2237183980 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e0c1a2550 con 0x7f7e0c07ee20 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.462+0000 7f7e09600700 1 -- 172.21.15.84:0/2237183980 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f7e0001a430 con 0x7f7e0c07ee20 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.462+0000 7f7e13400700 1 -- 172.21.15.84:0/2237183980 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e0c064a80 con 0x7f7e0c07ee20 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.462+0000 7f7e09600700 1 --2- 172.21.15.84:0/2237183980 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f7df4039600 0x7f7df403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.463+0000 7f7e09600700 1 -- 172.21.15.84:0/2237183980 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7e0004d990 con 0x7f7e0c07ee20 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.463+0000 7f7e0be00700 1 --2- 172.21.15.84:0/2237183980 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f7df4039600 0x7f7df403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.463+0000 7f7e0be00700 1 --2- 172.21.15.84:0/2237183980 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f7df4039600 0x7f7df403bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc006fd0 tx=0x7f7dfc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.468+0000 7f7e09600700 1 -- 172.21.15.84:0/2237183980 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7e00016260 con 0x7f7e0c07ee20 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.694+0000 7f7e13400700 1 -- 172.21.15.84:0/2237183980 --> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f7e0c1057b0 con 0x7f7df4039600 2024-04-28T12:40:03.412 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.695+0000 7f7e09600700 1 -- 172.21.15.84:0/2237183980 <== mgr.14120 v2:172.21.15.84:6800/4213070051 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f7e0c1057b0 con 0x7f7df4039600 2024-04-28T12:40:03.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.698+0000 7f7e13400700 1 -- 172.21.15.84:0/2237183980 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f7df4039600 msgr2=0x7f7df403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:03.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.698+0000 7f7e13400700 1 --2- 172.21.15.84:0/2237183980 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f7df4039600 0x7f7df403bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc006fd0 tx=0x7f7dfc006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:03.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.698+0000 7f7e13400700 1 -- 172.21.15.84:0/2237183980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 msgr2=0x7f7e0c19f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:03.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.698+0000 7f7e13400700 1 --2- 172.21.15.84:0/2237183980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 0x7f7e0c19f090 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7e00011ac0 tx=0x7f7e00011ba0 comp rx=0 tx=0).stop 2024-04-28T12:40:03.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.699+0000 7f7e13400700 1 -- 172.21.15.84:0/2237183980 shutdown_connections 2024-04-28T12:40:03.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.699+0000 7f7e13400700 1 --2- 172.21.15.84:0/2237183980 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f7df4039600 0x7f7df403bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:03.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.699+0000 7f7e13400700 1 --2- 172.21.15.84:0/2237183980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7e0c07ee20 0x7f7e0c19f090 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:03.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.699+0000 7f7e13400700 1 -- 172.21.15.84:0/2237183980 >> 172.21.15.84:0/2237183980 conn(0x7f7e0c104470 msgr2=0x7f7e0c1050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:03.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.699+0000 7f7e13400700 1 -- 172.21.15.84:0/2237183980 shutdown_connections 2024-04-28T12:40:03.413 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:02.699+0000 7f7e13400700 1 -- 172.21.15.84:0/2237183980 wait complete. 2024-04-28T12:40:03.413 INFO:teuthology.orchestra.run.smithi084.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-28T12:40:03.413 INFO:teuthology.orchestra.run.smithi084.stdout:Adding key to root@localhost authorized_keys... 2024-04-28T12:40:03.416 INFO:teuthology.orchestra.run.smithi084.stdout:Adding host smithi084... 2024-04-28T12:40:03.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:03 smithi084 ceph-mon[28081]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:40:04.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:04 smithi084 ceph-mon[28081]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi084", "addr": "172.21.15.84", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:40:05.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:05 smithi084 ceph-mon[28081]: Deploying cephadm binary to smithi084 2024-04-28T12:40:11.169 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Added host 'smithi084' with addr '172.21.15.84' 2024-04-28T12:40:11.169 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.025+0000 7f6444800700 1 Processor -- start 2024-04-28T12:40:11.169 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.025+0000 7f6444800700 1 -- start start 2024-04-28T12:40:11.169 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.025+0000 7f6444800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 0x7f643c109320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.025+0000 7f6444800700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f643c109860 con 0x7f643c108f00 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.026+0000 7f6441e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 0x7f643c109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.026+0000 7f6441e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 0x7f643c109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41242/0 (socket says 172.21.15.84:41242) 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.026+0000 7f6441e00700 1 -- 172.21.15.84:0/2537337808 learned_addr learned my addr 172.21.15.84:0/2537337808 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.026+0000 7f6441e00700 1 -- 172.21.15.84:0/2537337808 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643c10a060 con 0x7f643c108f00 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.027+0000 7f6441e00700 1 --2- 172.21.15.84:0/2537337808 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 0x7f643c109320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f64300098a0 tx=0x7f6430009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee7b90cc2aff5c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.027+0000 7f6440a00700 1 -- 172.21.15.84:0/2537337808 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6430004160 con 0x7f643c108f00 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.027+0000 7f6440a00700 1 -- 172.21.15.84:0/2537337808 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6430004300 con 0x7f643c108f00 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.027+0000 7f6440a00700 1 -- 172.21.15.84:0/2537337808 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6430011af0 con 0x7f643c108f00 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.028+0000 7f6444800700 1 -- 172.21.15.84:0/2537337808 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 msgr2=0x7f643c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.028+0000 7f6444800700 1 --2- 172.21.15.84:0/2537337808 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 0x7f643c109320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f64300098a0 tx=0x7f6430009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.028+0000 7f6444800700 1 -- 172.21.15.84:0/2537337808 shutdown_connections 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.028+0000 7f6444800700 1 --2- 172.21.15.84:0/2537337808 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 0x7f643c109320 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:11.170 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.028+0000 7f6444800700 1 -- 172.21.15.84:0/2537337808 >> 172.21.15.84:0/2537337808 conn(0x7f643c1044c0 msgr2=0x7f643c1068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.028+0000 7f6444800700 1 -- 172.21.15.84:0/2537337808 shutdown_connections 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.028+0000 7f6444800700 1 -- 172.21.15.84:0/2537337808 wait complete. 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.029+0000 7f6444800700 1 Processor -- start 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.029+0000 7f6444800700 1 -- start start 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.030+0000 7f6444800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 0x7f643c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.030+0000 7f6444800700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f643c10d4a0 con 0x7f643c108f00 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.030+0000 7f6441e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 0x7f643c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.030+0000 7f6441e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 0x7f643c19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41254/0 (socket says 172.21.15.84:41254) 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.030+0000 7f6441e00700 1 -- 172.21.15.84:0/3574837415 learned_addr learned my addr 172.21.15.84:0/3574837415 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.030+0000 7f6441e00700 1 -- 172.21.15.84:0/3574837415 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643c19f5f0 con 0x7f643c108f00 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.031+0000 7f6441e00700 1 --2- 172.21.15.84:0/3574837415 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 0x7f643c19f0b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6430011ac0 tx=0x7f6430011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.031+0000 7f6436a00700 1 -- 172.21.15.84:0/3574837415 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6430003970 con 0x7f643c108f00 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.031+0000 7f6436a00700 1 -- 172.21.15.84:0/3574837415 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6430003b10 con 0x7f643c108f00 2024-04-28T12:40:11.171 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.031+0000 7f6444800700 1 -- 172.21.15.84:0/3574837415 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f643c19f880 con 0x7f643c108f00 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.031+0000 7f6436a00700 1 -- 172.21.15.84:0/3574837415 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6430024610 con 0x7f643c108f00 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.031+0000 7f6444800700 1 -- 172.21.15.84:0/3574837415 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f643c1a2570 con 0x7f643c108f00 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.032+0000 7f6436a00700 1 -- 172.21.15.84:0/3574837415 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f643001a9e0 con 0x7f643c108f00 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.032+0000 7f6444800700 1 -- 172.21.15.84:0/3574837415 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f643c064a80 con 0x7f643c108f00 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.033+0000 7f6436a00700 1 --2- 172.21.15.84:0/3574837415 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f64240395b0 0x7f642403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.033+0000 7f6436a00700 1 -- 172.21.15.84:0/3574837415 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f643004da00 con 0x7f643c108f00 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.033+0000 7f6441400700 1 --2- 172.21.15.84:0/3574837415 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f64240395b0 0x7f642403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.034+0000 7f6441400700 1 --2- 172.21.15.84:0/3574837415 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f64240395b0 0x7f642403ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f642c006fd0 tx=0x7f642c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.038+0000 7f6436a00700 1 -- 172.21.15.84:0/3574837415 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6430019dd0 con 0x7f643c108f00 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:04.267+0000 7f6444800700 1 -- 172.21.15.84:0/3574837415 --> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi084", "addr": "172.21.15.84", "target": ["mon-mgr", ""]}) -- 0x7f643c105690 con 0x7f64240395b0 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:10.362+0000 7f6436a00700 1 -- 172.21.15.84:0/3574837415 <== mgr.14120 v2:172.21.15.84:6800/4213070051 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f643c105690 con 0x7f64240395b0 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:10.366+0000 7f6444800700 1 -- 172.21.15.84:0/3574837415 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f64240395b0 msgr2=0x7f642403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:10.366+0000 7f6444800700 1 --2- 172.21.15.84:0/3574837415 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f64240395b0 0x7f642403ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f642c006fd0 tx=0x7f642c006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:11.172 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:10.366+0000 7f6444800700 1 -- 172.21.15.84:0/3574837415 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 msgr2=0x7f643c19f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:11.173 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:10.366+0000 7f6444800700 1 --2- 172.21.15.84:0/3574837415 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 0x7f643c19f0b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6430011ac0 tx=0x7f6430011ba0 comp rx=0 tx=0).stop 2024-04-28T12:40:11.173 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:10.366+0000 7f6444800700 1 -- 172.21.15.84:0/3574837415 shutdown_connections 2024-04-28T12:40:11.173 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:10.366+0000 7f6444800700 1 --2- 172.21.15.84:0/3574837415 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f64240395b0 0x7f642403ba70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:11.173 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:10.366+0000 7f6444800700 1 --2- 172.21.15.84:0/3574837415 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f643c108f00 0x7f643c19f0b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:11.173 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:10.366+0000 7f6444800700 1 -- 172.21.15.84:0/3574837415 >> 172.21.15.84:0/3574837415 conn(0x7f643c1044c0 msgr2=0x7f643c104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:11.173 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:10.366+0000 7f6444800700 1 -- 172.21.15.84:0/3574837415 shutdown_connections 2024-04-28T12:40:11.173 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:10.366+0000 7f6444800700 1 -- 172.21.15.84:0/3574837415 wait complete. 2024-04-28T12:40:11.173 INFO:teuthology.orchestra.run.smithi084.stdout:Deploying unmanaged mon service... 2024-04-28T12:40:11.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:11 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:40:11.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:11 smithi084 ceph-mon[28081]: Added host smithi084 2024-04-28T12:40:11.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:11 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:40:12.693 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-28T12:40:12.693 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.745+0000 7fa2f1000700 1 Processor -- start 2024-04-28T12:40:12.693 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.745+0000 7fa2f1000700 1 -- start start 2024-04-28T12:40:12.693 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.746+0000 7fa2f1000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 0x7fa2ec07d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:12.693 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.746+0000 7fa2f1000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec07d8c0 con 0x7fa2ec07ee90 2024-04-28T12:40:12.693 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.746+0000 7fa2e9e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 0x7fa2ec07d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:12.693 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.746+0000 7fa2e9e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 0x7fa2ec07d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40192/0 (socket says 172.21.15.84:40192) 2024-04-28T12:40:12.693 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.746+0000 7fa2e9e00700 1 -- 172.21.15.84:0/2357722963 learned_addr learned my addr 172.21.15.84:0/2357722963 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:12.697 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.747+0000 7fa2e9e00700 1 -- 172.21.15.84:0/2357722963 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ec07da40 con 0x7fa2ec07ee90 2024-04-28T12:40:12.698 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.747+0000 7fa2e9e00700 1 --2- 172.21.15.84:0/2357722963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 0x7fa2ec07d2f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc0098a0 tx=0x7fa2dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ca6e11f0da6ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:12.698 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.747+0000 7fa2e8a00700 1 -- 172.21.15.84:0/2357722963 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2dc004160 con 0x7fa2ec07ee90 2024-04-28T12:40:12.698 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.748+0000 7fa2e8a00700 1 -- 172.21.15.84:0/2357722963 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa2dc004300 con 0x7fa2ec07ee90 2024-04-28T12:40:12.698 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.748+0000 7fa2e8a00700 1 -- 172.21.15.84:0/2357722963 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2dc011af0 con 0x7fa2ec07ee90 2024-04-28T12:40:12.698 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.748+0000 7fa2f1000700 1 -- 172.21.15.84:0/2357722963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 msgr2=0x7fa2ec07d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:12.698 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.748+0000 7fa2f1000700 1 --2- 172.21.15.84:0/2357722963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 0x7fa2ec07d2f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc0098a0 tx=0x7fa2dc009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:12.698 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.749+0000 7fa2f1000700 1 -- 172.21.15.84:0/2357722963 shutdown_connections 2024-04-28T12:40:12.698 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.749+0000 7fa2f1000700 1 --2- 172.21.15.84:0/2357722963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 0x7fa2ec07d2f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:12.698 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.749+0000 7fa2f1000700 1 -- 172.21.15.84:0/2357722963 >> 172.21.15.84:0/2357722963 conn(0x7fa2ec1044c0 msgr2=0x7fa2ec1068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:12.698 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.749+0000 7fa2f1000700 1 -- 172.21.15.84:0/2357722963 shutdown_connections 2024-04-28T12:40:12.698 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.749+0000 7fa2f1000700 1 -- 172.21.15.84:0/2357722963 wait complete. 2024-04-28T12:40:12.698 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.750+0000 7fa2f1000700 1 Processor -- start 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.750+0000 7fa2f1000700 1 -- start start 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.750+0000 7fa2f1000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 0x7fa2ec19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.750+0000 7fa2f1000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec10f4b0 con 0x7fa2ec07ee90 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.751+0000 7fa2e9e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 0x7fa2ec19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.751+0000 7fa2e9e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 0x7fa2ec19f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40196/0 (socket says 172.21.15.84:40196) 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.751+0000 7fa2e9e00700 1 -- 172.21.15.84:0/3372862465 learned_addr learned my addr 172.21.15.84:0/3372862465 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.751+0000 7fa2e9e00700 1 -- 172.21.15.84:0/3372862465 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ec19f610 con 0x7fa2ec07ee90 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.751+0000 7fa2e9e00700 1 --2- 172.21.15.84:0/3372862465 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 0x7fa2ec19f0d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc0117b0 tx=0x7fa2dc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.752+0000 7fa2e2a00700 1 -- 172.21.15.84:0/3372862465 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2dc003af0 con 0x7fa2ec07ee90 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.752+0000 7fa2e2a00700 1 -- 172.21.15.84:0/3372862465 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa2dc020070 con 0x7fa2ec07ee90 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.752+0000 7fa2e2a00700 1 -- 172.21.15.84:0/3372862465 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2dc019630 con 0x7fa2ec07ee90 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.752+0000 7fa2f1000700 1 -- 172.21.15.84:0/3372862465 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ec19f8a0 con 0x7fa2ec07ee90 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.752+0000 7fa2f1000700 1 -- 172.21.15.84:0/3372862465 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2ec1a2590 con 0x7fa2ec07ee90 2024-04-28T12:40:12.699 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.753+0000 7fa2f1000700 1 -- 172.21.15.84:0/3372862465 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2ec064a80 con 0x7fa2ec07ee90 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.753+0000 7fa2e2a00700 1 -- 172.21.15.84:0/3372862465 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fa2dc01a9c0 con 0x7fa2ec07ee90 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.754+0000 7fa2e2a00700 1 --2- 172.21.15.84:0/3372862465 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fa2d0039660 0x7fa2d003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.754+0000 7fa2e2a00700 1 -- 172.21.15.84:0/3372862465 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa2dc04dbf0 con 0x7fa2ec07ee90 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.754+0000 7fa2e9400700 1 --2- 172.21.15.84:0/3372862465 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fa2d0039660 0x7fa2d003bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.755+0000 7fa2e9400700 1 --2- 172.21.15.84:0/3372862465 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fa2d0039660 0x7fa2d003bb20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8006fd0 tx=0x7fa2d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:11.759+0000 7fa2e2a00700 1 -- 172.21.15.84:0/3372862465 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa2dc00e260 con 0x7fa2ec07ee90 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:12.001+0000 7fa2f1000700 1 -- 172.21.15.84:0/3372862465 --> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fa2ec1057f0 con 0x7fa2d0039660 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:12.012+0000 7fa2e2a00700 1 -- 172.21.15.84:0/3372862465 <== mgr.14120 v2:172.21.15.84:6800/4213070051 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa2ec1057f0 con 0x7fa2d0039660 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:12.015+0000 7fa2f1000700 1 -- 172.21.15.84:0/3372862465 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fa2d0039660 msgr2=0x7fa2d003bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:12.015+0000 7fa2f1000700 1 --2- 172.21.15.84:0/3372862465 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fa2d0039660 0x7fa2d003bb20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8006fd0 tx=0x7fa2d8006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:12.015+0000 7fa2f1000700 1 -- 172.21.15.84:0/3372862465 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 msgr2=0x7fa2ec19f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:12.015+0000 7fa2f1000700 1 --2- 172.21.15.84:0/3372862465 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 0x7fa2ec19f0d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc0117b0 tx=0x7fa2dc011b70 comp rx=0 tx=0).stop 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:12.015+0000 7fa2f1000700 1 -- 172.21.15.84:0/3372862465 shutdown_connections 2024-04-28T12:40:12.700 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:12.015+0000 7fa2f1000700 1 --2- 172.21.15.84:0/3372862465 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fa2d0039660 0x7fa2d003bb20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:12.701 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:12.015+0000 7fa2f1000700 1 --2- 172.21.15.84:0/3372862465 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2ec07ee90 0x7fa2ec19f0d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:12.701 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:12.015+0000 7fa2f1000700 1 -- 172.21.15.84:0/3372862465 >> 172.21.15.84:0/3372862465 conn(0x7fa2ec1044c0 msgr2=0x7fa2ec1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:12.701 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:12.015+0000 7fa2f1000700 1 -- 172.21.15.84:0/3372862465 shutdown_connections 2024-04-28T12:40:12.701 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:12.016+0000 7fa2f1000700 1 -- 172.21.15.84:0/3372862465 wait complete. 2024-04-28T12:40:12.701 INFO:teuthology.orchestra.run.smithi084.stdout:Deploying unmanaged mgr service... 2024-04-28T12:40:13.294 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:13 smithi084 ceph-mon[28081]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:40:13.295 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:13 smithi084 ceph-mon[28081]: Saving service mon spec with placement count:5 2024-04-28T12:40:13.295 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:13 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:40:14.382 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-28T12:40:14.382 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.372+0000 7f0f9ce00700 1 Processor -- start 2024-04-28T12:40:14.382 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.373+0000 7f0f9ce00700 1 -- start start 2024-04-28T12:40:14.382 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.373+0000 7f0f9ce00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 0x7f0f9807d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.373+0000 7f0f9ce00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9807d900 con 0x7f0f9807eed0 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.374+0000 7f0f95e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 0x7f0f9807d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.374+0000 7f0f95e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 0x7f0f9807d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40198/0 (socket says 172.21.15.84:40198) 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.374+0000 7f0f95e00700 1 -- 172.21.15.84:0/2364136387 learned_addr learned my addr 172.21.15.84:0/2364136387 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.374+0000 7f0f95e00700 1 -- 172.21.15.84:0/2364136387 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f9807da80 con 0x7f0f9807eed0 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.375+0000 7f0f95e00700 1 --2- 172.21.15.84:0/2364136387 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 0x7f0f9807d330 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0f880098a0 tx=0x7f0f88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ee2f3fe5082111 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.375+0000 7f0f95400700 1 -- 172.21.15.84:0/2364136387 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f88004160 con 0x7f0f9807eed0 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.375+0000 7f0f95400700 1 -- 172.21.15.84:0/2364136387 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0f88004300 con 0x7f0f9807eed0 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.375+0000 7f0f95400700 1 -- 172.21.15.84:0/2364136387 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f88011af0 con 0x7f0f9807eed0 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.375+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2364136387 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 msgr2=0x7f0f9807d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.375+0000 7f0f9ce00700 1 --2- 172.21.15.84:0/2364136387 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 0x7f0f9807d330 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0f880098a0 tx=0x7f0f88009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.376+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2364136387 shutdown_connections 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.376+0000 7f0f9ce00700 1 --2- 172.21.15.84:0/2364136387 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 0x7f0f9807d330 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:14.383 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.376+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2364136387 >> 172.21.15.84:0/2364136387 conn(0x7f0f981044f0 msgr2=0x7f0f98106910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.376+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2364136387 shutdown_connections 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.376+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2364136387 wait complete. 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.377+0000 7f0f9ce00700 1 Processor -- start 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.377+0000 7f0f9ce00700 1 -- start start 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.378+0000 7f0f9ce00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 0x7f0f981a34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.378+0000 7f0f9ce00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9810b0e0 con 0x7f0f9807eed0 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.378+0000 7f0f95e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 0x7f0f981a34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.378+0000 7f0f95e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 0x7f0f981a34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40212/0 (socket says 172.21.15.84:40212) 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.378+0000 7f0f95e00700 1 -- 172.21.15.84:0/2411408345 learned_addr learned my addr 172.21.15.84:0/2411408345 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.378+0000 7f0f95e00700 1 -- 172.21.15.84:0/2411408345 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f981a3a10 con 0x7f0f9807eed0 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.379+0000 7f0f95e00700 1 --2- 172.21.15.84:0/2411408345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 0x7f0f981a34d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009fb0 tx=0x7f0f88011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.379+0000 7f0f8ea00700 1 -- 172.21.15.84:0/2411408345 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f880036a0 con 0x7f0f9807eed0 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.379+0000 7f0f8ea00700 1 -- 172.21.15.84:0/2411408345 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0f8801a430 con 0x7f0f9807eed0 2024-04-28T12:40:14.384 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.379+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2411408345 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f981a3ca0 con 0x7f0f9807eed0 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.379+0000 7f0f8ea00700 1 -- 172.21.15.84:0/2411408345 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f88019550 con 0x7f0f9807eed0 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.379+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2411408345 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f981a6990 con 0x7f0f9807eed0 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.380+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2411408345 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f98064a80 con 0x7f0f9807eed0 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.380+0000 7f0f8ea00700 1 -- 172.21.15.84:0/2411408345 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0f8801a8e0 con 0x7f0f9807eed0 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.381+0000 7f0f8ea00700 1 --2- 172.21.15.84:0/2411408345 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f0f7c039560 0x7f0f7c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.381+0000 7f0f8ea00700 1 -- 172.21.15.84:0/2411408345 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0f8804d920 con 0x7f0f9807eed0 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.381+0000 7f0f8fe00700 1 --2- 172.21.15.84:0/2411408345 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f0f7c039560 0x7f0f7c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.382+0000 7f0f8fe00700 1 --2- 172.21.15.84:0/2411408345 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f0f7c039560 0x7f0f7c03ba20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0f80006fd0 tx=0x7f0f80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.386+0000 7f0f8ea00700 1 -- 172.21.15.84:0/2411408345 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0f8801a690 con 0x7f0f9807eed0 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.630+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2411408345 --> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f0f98075e40 con 0x7f0f7c039560 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.637+0000 7f0f8ea00700 1 -- 172.21.15.84:0/2411408345 <== mgr.14120 v2:172.21.15.84:6800/4213070051 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0f98075e40 con 0x7f0f7c039560 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.640+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2411408345 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f0f7c039560 msgr2=0x7f0f7c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.640+0000 7f0f9ce00700 1 --2- 172.21.15.84:0/2411408345 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f0f7c039560 0x7f0f7c03ba20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0f80006fd0 tx=0x7f0f80006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:14.385 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.640+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2411408345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 msgr2=0x7f0f981a34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:14.386 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.640+0000 7f0f9ce00700 1 --2- 172.21.15.84:0/2411408345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 0x7f0f981a34d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009fb0 tx=0x7f0f88011940 comp rx=0 tx=0).stop 2024-04-28T12:40:14.386 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.640+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2411408345 shutdown_connections 2024-04-28T12:40:14.386 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.640+0000 7f0f9ce00700 1 --2- 172.21.15.84:0/2411408345 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f0f7c039560 0x7f0f7c03ba20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:14.386 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.640+0000 7f0f9ce00700 1 --2- 172.21.15.84:0/2411408345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f9807eed0 0x7f0f981a34d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:14.386 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.640+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2411408345 >> 172.21.15.84:0/2411408345 conn(0x7f0f981044f0 msgr2=0x7f0f981068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:14.386 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.640+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2411408345 shutdown_connections 2024-04-28T12:40:14.386 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:13.640+0000 7f0f9ce00700 1 -- 172.21.15.84:0/2411408345 wait complete. 2024-04-28T12:40:14.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:14 smithi084 ceph-mon[28081]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:40:14.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:14 smithi084 ceph-mon[28081]: Saving service mgr spec with placement count:2 2024-04-28T12:40:14.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:14 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:40:16.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:16 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:40:16.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:16 smithi084 ceph-mon[28081]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:40:16.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:16 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:40:16.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:16 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/17093703' entity='client.admin' 2024-04-28T12:40:16.766 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.767+0000 7f4331200700 1 Processor -- start 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.767+0000 7f4331200700 1 -- start start 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.767+0000 7f4331200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 0x7f432c10bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.768+0000 7f4331200700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f432c07cea0 con 0x7f432c1099c0 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.768+0000 7f4329e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 0x7f432c10bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.769+0000 7f4329e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 0x7f432c10bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40218/0 (socket says 172.21.15.84:40218) 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.769+0000 7f4329e00700 1 -- 172.21.15.84:0/193537397 learned_addr learned my addr 172.21.15.84:0/193537397 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.769+0000 7f4329e00700 1 -- 172.21.15.84:0/193537397 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432c10c2f0 con 0x7f432c1099c0 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.769+0000 7f4329e00700 1 --2- 172.21.15.84:0/193537397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 0x7f432c10bdb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f431c00f960 tx=0x7f431c00fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cfc49d71ab70e97 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.770+0000 7f4328a00700 1 -- 172.21.15.84:0/193537397 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f431c01c070 con 0x7f432c1099c0 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.770+0000 7f4328a00700 1 -- 172.21.15.84:0/193537397 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f431c004030 con 0x7f432c1099c0 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.770+0000 7f4328a00700 1 -- 172.21.15.84:0/193537397 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f431c003c30 con 0x7f432c1099c0 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.770+0000 7f4331200700 1 -- 172.21.15.84:0/193537397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 msgr2=0x7f432c10bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.770+0000 7f4331200700 1 --2- 172.21.15.84:0/193537397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 0x7f432c10bdb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f431c00f960 tx=0x7f431c00fc70 comp rx=0 tx=0).stop 2024-04-28T12:40:16.767 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.771+0000 7f4331200700 1 -- 172.21.15.84:0/193537397 shutdown_connections 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.771+0000 7f4331200700 1 --2- 172.21.15.84:0/193537397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 0x7f432c10bdb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.771+0000 7f4331200700 1 -- 172.21.15.84:0/193537397 >> 172.21.15.84:0/193537397 conn(0x7f432c103410 msgr2=0x7f432c105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.771+0000 7f4331200700 1 -- 172.21.15.84:0/193537397 shutdown_connections 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.771+0000 7f4331200700 1 -- 172.21.15.84:0/193537397 wait complete. 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.772+0000 7f4331200700 1 Processor -- start 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.772+0000 7f4331200700 1 -- start start 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.772+0000 7f4331200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 0x7f432c19ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.772+0000 7f4331200700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f432c108590 con 0x7f432c1099c0 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.773+0000 7f4329e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 0x7f432c19ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.773+0000 7f4329e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 0x7f432c19ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40226/0 (socket says 172.21.15.84:40226) 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.773+0000 7f4329e00700 1 -- 172.21.15.84:0/17093703 learned_addr learned my addr 172.21.15.84:0/17093703 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.773+0000 7f4329e00700 1 -- 172.21.15.84:0/17093703 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432c19b190 con 0x7f432c1099c0 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.773+0000 7f4329e00700 1 --2- 172.21.15.84:0/17093703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 0x7f432c19ac50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f431c000c00 tx=0x7f431c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.774+0000 7f4322a00700 1 -- 172.21.15.84:0/17093703 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f431c025070 con 0x7f432c1099c0 2024-04-28T12:40:16.768 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.774+0000 7f4322a00700 1 -- 172.21.15.84:0/17093703 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f431c017900 con 0x7f432c1099c0 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.774+0000 7f4322a00700 1 -- 172.21.15.84:0/17093703 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f431c01c040 con 0x7f432c1099c0 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.774+0000 7f4331200700 1 -- 172.21.15.84:0/17093703 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f432c19b420 con 0x7f432c1099c0 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.774+0000 7f4331200700 1 -- 172.21.15.84:0/17093703 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f432c19e110 con 0x7f432c1099c0 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.775+0000 7f4322a00700 1 -- 172.21.15.84:0/17093703 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f431c028070 con 0x7f432c1099c0 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.775+0000 7f4322a00700 1 --2- 172.21.15.84:0/17093703 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f4310039660 0x7f431003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.776+0000 7f4322a00700 1 -- 172.21.15.84:0/17093703 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f431c014070 con 0x7f432c1099c0 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.776+0000 7f4331200700 1 -- 172.21.15.84:0/17093703 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f432c064a80 con 0x7f432c1099c0 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.776+0000 7f4329400700 1 --2- 172.21.15.84:0/17093703 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f4310039660 0x7f431003bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.777+0000 7f4329400700 1 --2- 172.21.15.84:0/17093703 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f4310039660 0x7f431003bb20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4318006fd0 tx=0x7f4318006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:15.782+0000 7f4322a00700 1 -- 172.21.15.84:0/17093703 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f431c01b170 con 0x7f432c1099c0 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:16.034+0000 7f4331200700 1 -- 172.21.15.84:0/17093703 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f432c02e620 con 0x7f432c1099c0 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:16.040+0000 7f4322a00700 1 -- 172.21.15.84:0/17093703 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f431c017db0 con 0x7f432c1099c0 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:16.043+0000 7f4331200700 1 -- 172.21.15.84:0/17093703 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f4310039660 msgr2=0x7f431003bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:16.769 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:16.043+0000 7f4331200700 1 --2- 172.21.15.84:0/17093703 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f4310039660 0x7f431003bb20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4318006fd0 tx=0x7f4318006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:16.770 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:16.043+0000 7f4331200700 1 -- 172.21.15.84:0/17093703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 msgr2=0x7f432c19ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:16.770 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:16.043+0000 7f4331200700 1 --2- 172.21.15.84:0/17093703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 0x7f432c19ac50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f431c000c00 tx=0x7f431c0044c0 comp rx=0 tx=0).stop 2024-04-28T12:40:16.770 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:16.043+0000 7f4331200700 1 -- 172.21.15.84:0/17093703 shutdown_connections 2024-04-28T12:40:16.770 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:16.043+0000 7f4331200700 1 --2- 172.21.15.84:0/17093703 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f4310039660 0x7f431003bb20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:16.770 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:16.043+0000 7f4331200700 1 --2- 172.21.15.84:0/17093703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f432c1099c0 0x7f432c19ac50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:16.770 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:16.043+0000 7f4331200700 1 -- 172.21.15.84:0/17093703 >> 172.21.15.84:0/17093703 conn(0x7f432c103410 msgr2=0x7f432c103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:16.770 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:16.044+0000 7f4331200700 1 -- 172.21.15.84:0/17093703 shutdown_connections 2024-04-28T12:40:16.770 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:16.044+0000 7f4331200700 1 -- 172.21.15.84:0/17093703 wait complete. 2024-04-28T12:40:18.436 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.398+0000 7f382ce00700 1 Processor -- start 2024-04-28T12:40:18.436 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.398+0000 7f382ce00700 1 -- start start 2024-04-28T12:40:18.436 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.399+0000 7f382ce00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 0x7f382807d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:18.436 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.399+0000 7f382ce00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f382807d810 con 0x7f382807ee20 2024-04-28T12:40:18.436 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.399+0000 7f3825e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 0x7f382807d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:18.436 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.399+0000 7f3825e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 0x7f382807d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40230/0 (socket says 172.21.15.84:40230) 2024-04-28T12:40:18.436 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.399+0000 7f3825e00700 1 -- 172.21.15.84:0/3714299930 learned_addr learned my addr 172.21.15.84:0/3714299930 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:18.436 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.399+0000 7f3825e00700 1 -- 172.21.15.84:0/3714299930 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382807d990 con 0x7f382807ee20 2024-04-28T12:40:18.437 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.400+0000 7f3825e00700 1 --2- 172.21.15.84:0/3714299930 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 0x7f382807d2d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f38180098a0 tx=0x7f3818009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237b16aa3e2e32d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:18.441 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.400+0000 7f3824a00700 1 -- 172.21.15.84:0/3714299930 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3818004160 con 0x7f382807ee20 2024-04-28T12:40:18.441 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.400+0000 7f3824a00700 1 -- 172.21.15.84:0/3714299930 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3818004300 con 0x7f382807ee20 2024-04-28T12:40:18.441 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.400+0000 7f3824a00700 1 -- 172.21.15.84:0/3714299930 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3818011a20 con 0x7f382807ee20 2024-04-28T12:40:18.441 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.401+0000 7f382ce00700 1 -- 172.21.15.84:0/3714299930 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 msgr2=0x7f382807d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:18.441 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.401+0000 7f382ce00700 1 --2- 172.21.15.84:0/3714299930 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 0x7f382807d2d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f38180098a0 tx=0x7f3818009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:18.441 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.401+0000 7f382ce00700 1 -- 172.21.15.84:0/3714299930 shutdown_connections 2024-04-28T12:40:18.441 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.401+0000 7f382ce00700 1 --2- 172.21.15.84:0/3714299930 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 0x7f382807d2d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:18.441 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.401+0000 7f382ce00700 1 -- 172.21.15.84:0/3714299930 >> 172.21.15.84:0/3714299930 conn(0x7f3828104470 msgr2=0x7f38281068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:18.441 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.401+0000 7f382ce00700 1 -- 172.21.15.84:0/3714299930 shutdown_connections 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.401+0000 7f382ce00700 1 -- 172.21.15.84:0/3714299930 wait complete. 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.402+0000 7f382ce00700 1 Processor -- start 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.402+0000 7f382ce00700 1 -- start start 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.403+0000 7f382ce00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 0x7f382819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.403+0000 7f382ce00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f382810f470 con 0x7f382807ee20 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.403+0000 7f3825e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 0x7f382819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.403+0000 7f3825e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 0x7f382819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40234/0 (socket says 172.21.15.84:40234) 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.403+0000 7f3825e00700 1 -- 172.21.15.84:0/2725963122 learned_addr learned my addr 172.21.15.84:0/2725963122 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.403+0000 7f3825e00700 1 -- 172.21.15.84:0/2725963122 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382819f550 con 0x7f382807ee20 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.404+0000 7f3825e00700 1 --2- 172.21.15.84:0/2725963122 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 0x7f382819f010 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38180116e0 tx=0x7f3818011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.404+0000 7f381ea00700 1 -- 172.21.15.84:0/2725963122 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3818003b20 con 0x7f382807ee20 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.404+0000 7f381ea00700 1 -- 172.21.15.84:0/2725963122 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f381801b8f0 con 0x7f382807ee20 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.404+0000 7f382ce00700 1 -- 172.21.15.84:0/2725963122 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f382819f7e0 con 0x7f382807ee20 2024-04-28T12:40:18.442 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.404+0000 7f381ea00700 1 -- 172.21.15.84:0/2725963122 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38180238c0 con 0x7f382807ee20 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.404+0000 7f382ce00700 1 -- 172.21.15.84:0/2725963122 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38281a24d0 con 0x7f382807ee20 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.405+0000 7f382ce00700 1 -- 172.21.15.84:0/2725963122 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3828064a80 con 0x7f382807ee20 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.405+0000 7f381ea00700 1 -- 172.21.15.84:0/2725963122 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f381801baa0 con 0x7f382807ee20 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.406+0000 7f381ea00700 1 --2- 172.21.15.84:0/2725963122 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f380c039560 0x7f380c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.406+0000 7f381ea00700 1 -- 172.21.15.84:0/2725963122 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f381804e140 con 0x7f382807ee20 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.406+0000 7f3825400700 1 --2- 172.21.15.84:0/2725963122 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f380c039560 0x7f380c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.407+0000 7f3825400700 1 --2- 172.21.15.84:0/2725963122 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f380c039560 0x7f380c03ba20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3814006fd0 tx=0x7f3814006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.412+0000 7f381ea00700 1 -- 172.21.15.84:0/2725963122 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f381801f030 con 0x7f382807ee20 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.650+0000 7f382ce00700 1 -- 172.21.15.84:0/2725963122 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f38281a26a0 con 0x7f382807ee20 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.655+0000 7f381ea00700 1 -- 172.21.15.84:0/2725963122 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f381801a3d0 con 0x7f382807ee20 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.659+0000 7f382ce00700 1 -- 172.21.15.84:0/2725963122 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f380c039560 msgr2=0x7f380c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.659+0000 7f382ce00700 1 --2- 172.21.15.84:0/2725963122 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f380c039560 0x7f380c03ba20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3814006fd0 tx=0x7f3814006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.659+0000 7f382ce00700 1 -- 172.21.15.84:0/2725963122 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 msgr2=0x7f382819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.659+0000 7f382ce00700 1 --2- 172.21.15.84:0/2725963122 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 0x7f382819f010 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38180116e0 tx=0x7f3818011aa0 comp rx=0 tx=0).stop 2024-04-28T12:40:18.443 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.659+0000 7f382ce00700 1 -- 172.21.15.84:0/2725963122 shutdown_connections 2024-04-28T12:40:18.444 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.659+0000 7f382ce00700 1 --2- 172.21.15.84:0/2725963122 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f380c039560 0x7f380c03ba20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:18.444 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.659+0000 7f382ce00700 1 --2- 172.21.15.84:0/2725963122 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f382807ee20 0x7f382819f010 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:18.444 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.659+0000 7f382ce00700 1 -- 172.21.15.84:0/2725963122 >> 172.21.15.84:0/2725963122 conn(0x7f3828104470 msgr2=0x7f3828105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:18.444 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.660+0000 7f382ce00700 1 -- 172.21.15.84:0/2725963122 shutdown_connections 2024-04-28T12:40:18.444 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:17.660+0000 7f382ce00700 1 -- 172.21.15.84:0/2725963122 wait complete. 2024-04-28T12:40:18.444 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling the dashboard module... 2024-04-28T12:40:18.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:18 smithi084 ceph-mon[28081]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:40:18.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:18 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2725963122' entity='client.admin' 2024-04-28T12:40:20.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:20 smithi084 ceph-mon[28081]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:40:20.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:20 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2468659220' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-28T12:40:20.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:20 smithi084 ceph-mon[28081]: from='mgr.14120 172.21.15.84:0/289658111' entity='mgr.x' 2024-04-28T12:40:20.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:20 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: ignoring --setuser ceph since I am not root 2024-04-28T12:40:20.668 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:20 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: ignoring --setgroup ceph since I am not root 2024-04-28T12:40:21.008 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:20 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:20.744+0000 7f021d19b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T12:40:21.009 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:21 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:21.011+0000 7f021d19b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:40:21.213 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.196+0000 7f02c4c00700 1 Processor -- start 2024-04-28T12:40:21.214 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.196+0000 7f02c4c00700 1 -- start start 2024-04-28T12:40:21.214 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.197+0000 7f02c4c00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 0x7f02c01120b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:21.214 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.197+0000 7f02c4c00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f02c0112680 con 0x7f02c010fcc0 2024-04-28T12:40:21.214 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.197+0000 7f02bde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 0x7f02c01120b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:21.214 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.197+0000 7f02bde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 0x7f02c01120b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41818/0 (socket says 172.21.15.84:41818) 2024-04-28T12:40:21.214 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.197+0000 7f02bde00700 1 -- 172.21.15.84:0/4199506977 learned_addr learned my addr 172.21.15.84:0/4199506977 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:21.214 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.198+0000 7f02bde00700 1 -- 172.21.15.84:0/4199506977 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c0112800 con 0x7f02c010fcc0 2024-04-28T12:40:21.214 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.198+0000 7f02bde00700 1 --2- 172.21.15.84:0/4199506977 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 0x7f02c01120b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02b00098a0 tx=0x7f02b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5337a1e405edd3dd server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:21.214 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.198+0000 7f02bca00700 1 -- 172.21.15.84:0/4199506977 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02b0004160 con 0x7f02c010fcc0 2024-04-28T12:40:21.214 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.198+0000 7f02bca00700 1 -- 172.21.15.84:0/4199506977 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02b0004300 con 0x7f02c010fcc0 2024-04-28T12:40:21.214 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.198+0000 7f02bca00700 1 -- 172.21.15.84:0/4199506977 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02b0011a20 con 0x7f02c010fcc0 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.199+0000 7f02c4c00700 1 -- 172.21.15.84:0/4199506977 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 msgr2=0x7f02c01120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.199+0000 7f02c4c00700 1 --2- 172.21.15.84:0/4199506977 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 0x7f02c01120b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02b00098a0 tx=0x7f02b0009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.199+0000 7f02c4c00700 1 -- 172.21.15.84:0/4199506977 shutdown_connections 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.199+0000 7f02c4c00700 1 --2- 172.21.15.84:0/4199506977 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 0x7f02c01120b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.199+0000 7f02c4c00700 1 -- 172.21.15.84:0/4199506977 >> 172.21.15.84:0/4199506977 conn(0x7f02c007e540 msgr2=0x7f02c0080960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.199+0000 7f02c4c00700 1 -- 172.21.15.84:0/4199506977 shutdown_connections 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.199+0000 7f02c4c00700 1 -- 172.21.15.84:0/4199506977 wait complete. 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.200+0000 7f02c4c00700 1 Processor -- start 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.200+0000 7f02c4c00700 1 -- start start 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.201+0000 7f02c4c00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 0x7f02c01a3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.201+0000 7f02c4c00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f02c01138c0 con 0x7f02c010fcc0 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.201+0000 7f02bde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 0x7f02c01a3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.201+0000 7f02bde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 0x7f02c01a3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41834/0 (socket says 172.21.15.84:41834) 2024-04-28T12:40:21.215 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.201+0000 7f02bde00700 1 -- 172.21.15.84:0/2468659220 learned_addr learned my addr 172.21.15.84:0/2468659220 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.201+0000 7f02bde00700 1 -- 172.21.15.84:0/2468659220 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c01a38c0 con 0x7f02c010fcc0 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.201+0000 7f02bde00700 1 --2- 172.21.15.84:0/2468659220 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 0x7f02c01a3380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f02b0004660 tx=0x7f02b0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.202+0000 7f02b6a00700 1 -- 172.21.15.84:0/2468659220 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02b0003aa0 con 0x7f02c010fcc0 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.202+0000 7f02b6a00700 1 -- 172.21.15.84:0/2468659220 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02b001b8f0 con 0x7f02c010fcc0 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.202+0000 7f02b6a00700 1 -- 172.21.15.84:0/2468659220 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02b0023900 con 0x7f02c010fcc0 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.202+0000 7f02c4c00700 1 -- 172.21.15.84:0/2468659220 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c01a3b50 con 0x7f02c010fcc0 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.202+0000 7f02c4c00700 1 -- 172.21.15.84:0/2468659220 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c01a6840 con 0x7f02c010fcc0 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.203+0000 7f02b6a00700 1 -- 172.21.15.84:0/2468659220 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f02b001baa0 con 0x7f02c010fcc0 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.203+0000 7f02c4c00700 1 -- 172.21.15.84:0/2468659220 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02c0064a80 con 0x7f02c010fcc0 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.203+0000 7f02b6a00700 1 --2- 172.21.15.84:0/2468659220 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f02a4039600 0x7f02a403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.203+0000 7f02b6a00700 1 -- 172.21.15.84:0/2468659220 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f02b004df80 con 0x7f02c010fcc0 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.204+0000 7f02bd400700 1 --2- 172.21.15.84:0/2468659220 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f02a4039600 0x7f02a403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.204+0000 7f02bd400700 1 --2- 172.21.15.84:0/2468659220 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f02a4039600 0x7f02a403bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f02ac006fd0 tx=0x7f02ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.209+0000 7f02b6a00700 1 -- 172.21.15.84:0/2468659220 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f02b001f030 con 0x7f02c010fcc0 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:19.506+0000 7f02c4c00700 1 -- 172.21.15.84:0/2468659220 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f02c01a6ec0 con 0x7f02c010fcc0 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:20.388+0000 7f02b6a00700 1 -- 172.21.15.84:0/2468659220 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f02b00202b0 con 0x7f02c010fcc0 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:20.391+0000 7f02b6a00700 1 -- 172.21.15.84:0/2468659220 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f02b002c3f0 con 0x7f02c010fcc0 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:20.393+0000 7f02c4c00700 1 -- 172.21.15.84:0/2468659220 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f02a4039600 msgr2=0x7f02a403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:20.393+0000 7f02c4c00700 1 --2- 172.21.15.84:0/2468659220 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f02a4039600 0x7f02a403bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f02ac006fd0 tx=0x7f02ac006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:20.393+0000 7f02c4c00700 1 -- 172.21.15.84:0/2468659220 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 msgr2=0x7f02c01a3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:20.393+0000 7f02c4c00700 1 --2- 172.21.15.84:0/2468659220 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 0x7f02c01a3380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f02b0004660 tx=0x7f02b0011aa0 comp rx=0 tx=0).stop 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:20.393+0000 7f02c4c00700 1 -- 172.21.15.84:0/2468659220 shutdown_connections 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:20.393+0000 7f02c4c00700 1 --2- 172.21.15.84:0/2468659220 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7f02a4039600 0x7f02a403bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:20.393+0000 7f02c4c00700 1 --2- 172.21.15.84:0/2468659220 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f02c010fcc0 0x7f02c01a3380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:20.393+0000 7f02c4c00700 1 -- 172.21.15.84:0/2468659220 >> 172.21.15.84:0/2468659220 conn(0x7f02c007e540 msgr2=0x7f02c007f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:20.395+0000 7f02c4c00700 1 -- 172.21.15.84:0/2468659220 shutdown_connections 2024-04-28T12:40:21.217 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:20.395+0000 7f02c4c00700 1 -- 172.21.15.84:0/2468659220 wait complete. 2024-04-28T12:40:21.368 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:21 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:21.148+0000 7f021d19b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T12:40:21.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:21 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2468659220' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-28T12:40:21.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:21 smithi084 ceph-mon[28081]: mgrmap e9: x(active, since 25s) 2024-04-28T12:40:21.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:21 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:21.563+0000 7f021d19b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T12:40:22.079 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:21 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:21.677+0000 7f021d19b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T12:40:22.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:22 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1261968614' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-28T12:40:22.909 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:40:22.909 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-28T12:40:22.909 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-28T12:40:22.909 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-28T12:40:22.909 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-28T12:40:22.909 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:40:22.909 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.873+0000 7fbdbee00700 1 Processor -- start 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.873+0000 7fbdbee00700 1 -- start start 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.873+0000 7fbdbee00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 0x7fbdb81092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.873+0000 7fbdbee00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb8109820 con 0x7fbdb8108ec0 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.874+0000 7fbdb7e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 0x7fbdb81092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.874+0000 7fbdb7e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 0x7fbdb81092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41856/0 (socket says 172.21.15.84:41856) 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.874+0000 7fbdb7e00700 1 -- 172.21.15.84:0/2113298971 learned_addr learned my addr 172.21.15.84:0/2113298971 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.874+0000 7fbdb7e00700 1 -- 172.21.15.84:0/2113298971 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb810a060 con 0x7fbdb8108ec0 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.875+0000 7fbdb7e00700 1 --2- 172.21.15.84:0/2113298971 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 0x7fbdb81092e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbda80098a0 tx=0x7fbda8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9795f7f20f3ce8fc server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.875+0000 7fbdb6a00700 1 -- 172.21.15.84:0/2113298971 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbda8004160 con 0x7fbdb8108ec0 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.875+0000 7fbdb6a00700 1 -- 172.21.15.84:0/2113298971 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbda8004300 con 0x7fbdb8108ec0 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.875+0000 7fbdb6a00700 1 -- 172.21.15.84:0/2113298971 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbda8011a20 con 0x7fbdb8108ec0 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.876+0000 7fbdbee00700 1 -- 172.21.15.84:0/2113298971 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 msgr2=0x7fbdb81092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:22.910 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.876+0000 7fbdbee00700 1 --2- 172.21.15.84:0/2113298971 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 0x7fbdb81092e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbda80098a0 tx=0x7fbda8009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.876+0000 7fbdbee00700 1 -- 172.21.15.84:0/2113298971 shutdown_connections 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.876+0000 7fbdbee00700 1 --2- 172.21.15.84:0/2113298971 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 0x7fbdb81092e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.876+0000 7fbdbee00700 1 -- 172.21.15.84:0/2113298971 >> 172.21.15.84:0/2113298971 conn(0x7fbdb8104470 msgr2=0x7fbdb81068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.876+0000 7fbdbee00700 1 -- 172.21.15.84:0/2113298971 shutdown_connections 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.876+0000 7fbdbee00700 1 -- 172.21.15.84:0/2113298971 wait complete. 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.877+0000 7fbdbee00700 1 Processor -- start 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.877+0000 7fbdbee00700 1 -- start start 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.878+0000 7fbdbee00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 0x7fbdb807eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.878+0000 7fbdbee00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb810d320 con 0x7fbdb8108ec0 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.878+0000 7fbdb7e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 0x7fbdb807eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.878+0000 7fbdb7e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 0x7fbdb807eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41860/0 (socket says 172.21.15.84:41860) 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.878+0000 7fbdb7e00700 1 -- 172.21.15.84:0/1261968614 learned_addr learned my addr 172.21.15.84:0/1261968614 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.878+0000 7fbdb7e00700 1 -- 172.21.15.84:0/1261968614 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb807f0a0 con 0x7fbdb8108ec0 2024-04-28T12:40:22.911 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.879+0000 7fbdb7e00700 1 --2- 172.21.15.84:0/1261968614 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 0x7fbdb807eb60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbda80119f0 tx=0x7fbda8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.879+0000 7fbdb4c00700 1 -- 172.21.15.84:0/1261968614 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbda80039e0 con 0x7fbdb8108ec0 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.879+0000 7fbdb4c00700 1 -- 172.21.15.84:0/1261968614 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbda801b8f0 con 0x7fbdb8108ec0 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.879+0000 7fbdb4c00700 1 -- 172.21.15.84:0/1261968614 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbda80237a0 con 0x7fbdb8108ec0 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.879+0000 7fbdbee00700 1 -- 172.21.15.84:0/1261968614 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdb807d290 con 0x7fbdb8108ec0 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.879+0000 7fbdbee00700 1 -- 172.21.15.84:0/1261968614 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdb807d7d0 con 0x7fbdb8108ec0 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.880+0000 7fbdb4c00700 1 -- 172.21.15.84:0/1261968614 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fbda801b440 con 0x7fbdb8108ec0 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.881+0000 7fbdbee00700 1 -- 172.21.15.84:0/1261968614 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdb8064a80 con 0x7fbdb8108ec0 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.881+0000 7fbdb4c00700 1 --2- 172.21.15.84:0/1261968614 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbd9c0395b0 0x7fbd9c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.881+0000 7fbdb4c00700 1 -- 172.21.15.84:0/1261968614 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbda804df40 con 0x7fbdb8108ec0 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.881+0000 7fbdb7400700 1 -- 172.21.15.84:0/1261968614 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbd9c0395b0 msgr2=0x7fbd9c03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4213070051 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.881+0000 7fbdb7400700 1 --2- 172.21.15.84:0/1261968614 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbd9c0395b0 0x7fbd9c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:21.886+0000 7fbdb4c00700 1 -- 172.21.15.84:0/1261968614 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbda801f030 con 0x7fbdb8108ec0 2024-04-28T12:40:22.912 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.082+0000 7fbdb7400700 1 -- 172.21.15.84:0/1261968614 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbd9c0395b0 msgr2=0x7fbd9c03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4213070051 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.082+0000 7fbdb7400700 1 --2- 172.21.15.84:0/1261968614 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbd9c0395b0 0x7fbd9c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.167+0000 7fbdbee00700 1 -- 172.21.15.84:0/1261968614 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fbdb807db10 con 0x7fbdb8108ec0 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.167+0000 7fbdb4c00700 1 -- 172.21.15.84:0/1261968614 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fbdb807db10 con 0x7fbdb8108ec0 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.170+0000 7fbdbee00700 1 -- 172.21.15.84:0/1261968614 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbd9c0395b0 msgr2=0x7fbd9c03ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.170+0000 7fbdbee00700 1 --2- 172.21.15.84:0/1261968614 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbd9c0395b0 0x7fbd9c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.170+0000 7fbdbee00700 1 -- 172.21.15.84:0/1261968614 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 msgr2=0x7fbdb807eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.170+0000 7fbdbee00700 1 --2- 172.21.15.84:0/1261968614 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 0x7fbdb807eb60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbda80119f0 tx=0x7fbda8011ad0 comp rx=0 tx=0).stop 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.170+0000 7fbdbee00700 1 -- 172.21.15.84:0/1261968614 shutdown_connections 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.170+0000 7fbdbee00700 1 --2- 172.21.15.84:0/1261968614 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fbd9c0395b0 0x7fbd9c03ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.170+0000 7fbdbee00700 1 --2- 172.21.15.84:0/1261968614 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbdb8108ec0 0x7fbdb807eb60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.170+0000 7fbdbee00700 1 -- 172.21.15.84:0/1261968614 >> 172.21.15.84:0/1261968614 conn(0x7fbdb8104470 msgr2=0x7fbdb8105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.170+0000 7fbdbee00700 1 -- 172.21.15.84:0/1261968614 shutdown_connections 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:22.170+0000 7fbdbee00700 1 -- 172.21.15.84:0/1261968614 wait complete. 2024-04-28T12:40:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for the mgr to restart... 2024-04-28T12:40:22.914 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr epoch 9... 2024-04-28T12:40:23.473 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:23 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:23.278+0000 7f021d19b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T12:40:23.726 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:23 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:23.511+0000 7f021d19b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T12:40:23.726 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:23 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:23.728+0000 7f021d19b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T12:40:24.417 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:24 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:24.106+0000 7f021d19b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T12:40:24.417 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:24 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:24.230+0000 7f021d19b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T12:40:24.733 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:24 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:24.484+0000 7f021d19b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T12:40:24.733 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:24 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:24.735+0000 7f021d19b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:40:25.917 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:25 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:25.531+0000 7f021d19b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T12:40:26.417 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:26 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:26.078+0000 7f021d19b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T12:40:26.417 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:26 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:26.207+0000 7f021d19b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T12:40:26.917 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:26 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:26.473+0000 7f021d19b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T12:40:27.417 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:27 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:27.116+0000 7f021d19b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T12:40:27.417 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:27 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:27.240+0000 7f021d19b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T12:40:27.862 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:27 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:27.561+0000 7f021d19b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T12:40:27.862 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:27 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:27.738+0000 7f021d19b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T12:40:28.167 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:27 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:27.865+0000 7f021d19b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T12:40:28.167 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:28 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:28.016+0000 7f021d19b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T12:40:28.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:28 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:28.386+0000 7f021d19b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T12:40:29.417 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:29 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:29.157+0000 7f021d19b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T12:40:29.917 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:40:29 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:40:29.441+0000 7f021d19b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T12:40:29.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:29 smithi084 ceph-mon[28081]: Active manager daemon x restarted 2024-04-28T12:40:29.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:29 smithi084 ceph-mon[28081]: Activating manager daemon x 2024-04-28T12:40:29.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:29 smithi084 ceph-mon[28081]: osdmap e3: 0 total, 0 up, 0 in 2024-04-28T12:40:29.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:29 smithi084 ceph-mon[28081]: mgrmap e10: x(active, starting, since 0.00941337s) 2024-04-28T12:40:29.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:40:29.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-28T12:40:29.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-28T12:40:29.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-28T12:40:29.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-28T12:40:29.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:29 smithi084 ceph-mon[28081]: Manager daemon x is now available 2024-04-28T12:40:30.789 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:40:30.789 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-28T12:40:30.789 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-28T12:40:30.789 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:30 smithi084 ceph-mon[28081]: mgrmap e11: x(active, since 1.00998s) 2024-04-28T12:40:31.510 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-04-28T12:40:31.511 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-28T12:40:31.511 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-28T12:40:31.511 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-04-28T12:40:31.512 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.532+0000 7fb64d000700 1 Processor -- start 2024-04-28T12:40:31.512 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.532+0000 7fb64d000700 1 -- start start 2024-04-28T12:40:31.512 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.532+0000 7fb64d000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 0x7fb64807d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:31.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.532+0000 7fb64d000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb64807d7f0 con 0x7fb64807ee00 2024-04-28T12:40:31.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.533+0000 7fb647400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 0x7fb64807d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:31.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.533+0000 7fb647400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 0x7fb64807d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41868/0 (socket says 172.21.15.84:41868) 2024-04-28T12:40:31.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.533+0000 7fb647400700 1 -- 172.21.15.84:0/835294948 learned_addr learned my addr 172.21.15.84:0/835294948 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:31.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.533+0000 7fb647400700 1 -- 172.21.15.84:0/835294948 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64807d970 con 0x7fb64807ee00 2024-04-28T12:40:31.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.534+0000 7fb647400700 1 --2- 172.21.15.84:0/835294948 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 0x7fb64807d2b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb6380098a0 tx=0x7fb638009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e760b61f5f2a416e server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:31.513 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.534+0000 7fb646000700 1 -- 172.21.15.84:0/835294948 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb638004160 con 0x7fb64807ee00 2024-04-28T12:40:31.514 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.534+0000 7fb646000700 1 -- 172.21.15.84:0/835294948 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb638004300 con 0x7fb64807ee00 2024-04-28T12:40:31.514 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.534+0000 7fb646000700 1 -- 172.21.15.84:0/835294948 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb638011a20 con 0x7fb64807ee00 2024-04-28T12:40:31.516 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.535+0000 7fb64d000700 1 -- 172.21.15.84:0/835294948 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 msgr2=0x7fb64807d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:31.516 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.535+0000 7fb64d000700 1 --2- 172.21.15.84:0/835294948 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 0x7fb64807d2b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb6380098a0 tx=0x7fb638009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:31.516 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.535+0000 7fb64d000700 1 -- 172.21.15.84:0/835294948 shutdown_connections 2024-04-28T12:40:31.516 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.535+0000 7fb64d000700 1 --2- 172.21.15.84:0/835294948 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 0x7fb64807d2b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:31.516 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.535+0000 7fb64d000700 1 -- 172.21.15.84:0/835294948 >> 172.21.15.84:0/835294948 conn(0x7fb6481042d0 msgr2=0x7fb648106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:31.516 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.535+0000 7fb64d000700 1 -- 172.21.15.84:0/835294948 shutdown_connections 2024-04-28T12:40:31.516 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.535+0000 7fb64d000700 1 -- 172.21.15.84:0/835294948 wait complete. 2024-04-28T12:40:31.516 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.536+0000 7fb64d000700 1 Processor -- start 2024-04-28T12:40:31.516 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.536+0000 7fb64d000700 1 -- start start 2024-04-28T12:40:31.516 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.536+0000 7fb64d000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 0x7fb64819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:31.517 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.536+0000 7fb64d000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb64810f2d0 con 0x7fb64807ee00 2024-04-28T12:40:31.517 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.537+0000 7fb647400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 0x7fb64819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:31.517 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.537+0000 7fb647400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 0x7fb64819ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41872/0 (socket says 172.21.15.84:41872) 2024-04-28T12:40:31.517 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.537+0000 7fb647400700 1 -- 172.21.15.84:0/3244795462 learned_addr learned my addr 172.21.15.84:0/3244795462 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:31.517 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.537+0000 7fb647400700 1 -- 172.21.15.84:0/3244795462 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64819f3c0 con 0x7fb64807ee00 2024-04-28T12:40:31.517 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.537+0000 7fb647400700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 0x7fb64819ee80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb638004660 tx=0x7fb638011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:31.517 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.538+0000 7fb63fe00700 1 -- 172.21.15.84:0/3244795462 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb638003aa0 con 0x7fb64807ee00 2024-04-28T12:40:31.517 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.538+0000 7fb63fe00700 1 -- 172.21.15.84:0/3244795462 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb63801b8f0 con 0x7fb64807ee00 2024-04-28T12:40:31.517 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.538+0000 7fb64d000700 1 -- 172.21.15.84:0/3244795462 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb64819f650 con 0x7fb64807ee00 2024-04-28T12:40:31.517 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.538+0000 7fb63fe00700 1 -- 172.21.15.84:0/3244795462 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb638023900 con 0x7fb64807ee00 2024-04-28T12:40:31.520 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.538+0000 7fb64d000700 1 -- 172.21.15.84:0/3244795462 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6481a2340 con 0x7fb64807ee00 2024-04-28T12:40:31.520 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.539+0000 7fb63fe00700 1 -- 172.21.15.84:0/3244795462 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fb63801baa0 con 0x7fb64807ee00 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.539+0000 7fb63fe00700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 0x7fb62c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.539+0000 7fb63fe00700 1 -- 172.21.15.84:0/3244795462 --> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb62c03c1d0 con 0x7fb62c039600 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.539+0000 7fb63fe00700 1 -- 172.21.15.84:0/3244795462 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb63804dff0 con 0x7fb64807ee00 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.540+0000 7fb646a00700 1 -- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 msgr2=0x7fb62c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4213070051 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.540+0000 7fb646a00700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 0x7fb62c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.741+0000 7fb646a00700 1 -- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 msgr2=0x7fb62c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4213070051 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:23.741+0000 7fb646a00700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 0x7fb62c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:24.142+0000 7fb646a00700 1 -- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 msgr2=0x7fb62c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4213070051 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:24.142+0000 7fb646a00700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 0x7fb62c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:24.943+0000 7fb646a00700 1 -- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 msgr2=0x7fb62c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4213070051 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:24.943+0000 7fb646a00700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 0x7fb62c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:26.545+0000 7fb646a00700 1 -- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 msgr2=0x7fb62c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/4213070051 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:26.545+0000 7fb646a00700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 0x7fb62c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-28T12:40:31.521 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:29.451+0000 7fb63fe00700 1 -- 172.21.15.84:0/3244795462 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7fb63802c3f0 con 0x7fb64807ee00 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:29.451+0000 7fb63fe00700 1 -- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 msgr2=0x7fb62c03bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:29.451+0000 7fb63fe00700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 0x7fb62c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.452+0000 7fb63fe00700 1 -- 172.21.15.84:0/3244795462 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7fb638050070 con 0x7fb64807ee00 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.452+0000 7fb63fe00700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb62c03cfe0 0x7fb62c03f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.452+0000 7fb63fe00700 1 -- 172.21.15.84:0/3244795462 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb638017ec0 con 0x7fb62c03cfe0 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.452+0000 7fb646a00700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb62c03cfe0 0x7fb62c03f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.454+0000 7fb646a00700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb62c03cfe0 0x7fb62c03f3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb634003a10 tx=0x7fb6340092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.455+0000 7fb63fe00700 1 -- 172.21.15.84:0/3244795462 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fb638017ec0 con 0x7fb62c03cfe0 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.464+0000 7fb64d000700 1 -- 172.21.15.84:0/3244795462 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fb64806a590 con 0x7fb62c03cfe0 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.465+0000 7fb63fe00700 1 -- 172.21.15.84:0/3244795462 <== mgr.14152 v2:172.21.15.84:6800/2396287759 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fb64806a590 con 0x7fb62c03cfe0 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.465+0000 7fb64d000700 1 -- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb62c03cfe0 msgr2=0x7fb62c03f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.465+0000 7fb64d000700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb62c03cfe0 0x7fb62c03f3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb634003a10 tx=0x7fb6340092b0 comp rx=0 tx=0).stop 2024-04-28T12:40:31.522 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.465+0000 7fb64d000700 1 -- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 msgr2=0x7fb64819ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:31.523 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.465+0000 7fb64d000700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 0x7fb64819ee80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb638004660 tx=0x7fb638011aa0 comp rx=0 tx=0).stop 2024-04-28T12:40:31.524 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.465+0000 7fb64d000700 1 -- 172.21.15.84:0/3244795462 shutdown_connections 2024-04-28T12:40:31.524 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.465+0000 7fb64d000700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb64807ee00 0x7fb64819ee80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:31.524 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.465+0000 7fb64d000700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb62c03cfe0 0x7fb62c03f3d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:31.524 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.465+0000 7fb64d000700 1 --2- 172.21.15.84:0/3244795462 >> [v2:172.21.15.84:6800/4213070051,v1:172.21.15.84:6801/4213070051] conn(0x7fb62c039600 0x7fb62c03bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:31.524 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.465+0000 7fb64d000700 1 -- 172.21.15.84:0/3244795462 >> 172.21.15.84:0/3244795462 conn(0x7fb6481042d0 msgr2=0x7fb648104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:31.524 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.466+0000 7fb64d000700 1 -- 172.21.15.84:0/3244795462 shutdown_connections 2024-04-28T12:40:31.524 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:30.466+0000 7fb64d000700 1 -- 172.21.15.84:0/3244795462 wait complete. 2024-04-28T12:40:31.524 INFO:teuthology.orchestra.run.smithi084.stdout:mgr epoch 9 is available 2024-04-28T12:40:31.524 INFO:teuthology.orchestra.run.smithi084.stdout:Generating a dashboard self-signed certificate... 2024-04-28T12:40:31.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:31 smithi084 ceph-mon[28081]: [28/Apr/2024:12:40:30] ENGINE Bus STARTING 2024-04-28T12:40:31.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:31 smithi084 ceph-mon[28081]: [28/Apr/2024:12:40:30] ENGINE Serving on https://172.21.15.84:7150 2024-04-28T12:40:31.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:31 smithi084 ceph-mon[28081]: [28/Apr/2024:12:40:30] ENGINE Serving on http://172.21.15.84:8765 2024-04-28T12:40:31.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:31 smithi084 ceph-mon[28081]: [28/Apr/2024:12:40:30] ENGINE Bus STARTED 2024-04-28T12:40:31.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:31 smithi084 ceph-mon[28081]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-28T12:40:31.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:31 smithi084 ceph-mon[28081]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-28T12:40:32.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:32 smithi084 ceph-mon[28081]: mgrmap e12: x(active, since 2s) 2024-04-28T12:40:32.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:33.405 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-28T12:40:33.405 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.336+0000 7f48caa00700 1 Processor -- start 2024-04-28T12:40:33.405 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.337+0000 7f48caa00700 1 -- start start 2024-04-28T12:40:33.405 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.337+0000 7f48caa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 0x7f48c41092c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:33.405 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.337+0000 7f48caa00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f48c4109800 con 0x7f48c4108ea0 2024-04-28T12:40:33.406 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.337+0000 7f48c3e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 0x7f48c41092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:33.406 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.338+0000 7f48c3e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 0x7f48c41092c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37132/0 (socket says 172.21.15.84:37132) 2024-04-28T12:40:33.406 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.338+0000 7f48c3e00700 1 -- 172.21.15.84:0/3989547005 learned_addr learned my addr 172.21.15.84:0/3989547005 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:33.406 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.338+0000 7f48c3e00700 1 -- 172.21.15.84:0/3989547005 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c410a090 con 0x7f48c4108ea0 2024-04-28T12:40:33.406 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.338+0000 7f48c3e00700 1 --2- 172.21.15.84:0/3989547005 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 0x7f48c41092c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f48b40098a0 tx=0x7f48b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81cccbacd16e760c server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:33.406 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.339+0000 7f48c2a00700 1 -- 172.21.15.84:0/3989547005 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48b4004160 con 0x7f48c4108ea0 2024-04-28T12:40:33.406 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.339+0000 7f48c2a00700 1 -- 172.21.15.84:0/3989547005 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b4004300 con 0x7f48c4108ea0 2024-04-28T12:40:33.406 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.339+0000 7f48c2a00700 1 -- 172.21.15.84:0/3989547005 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48b4011a20 con 0x7f48c4108ea0 2024-04-28T12:40:33.406 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.339+0000 7f48caa00700 1 -- 172.21.15.84:0/3989547005 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 msgr2=0x7f48c41092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:33.406 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.339+0000 7f48caa00700 1 --2- 172.21.15.84:0/3989547005 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 0x7f48c41092c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f48b40098a0 tx=0x7f48b4009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:33.406 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.339+0000 7f48caa00700 1 -- 172.21.15.84:0/3989547005 shutdown_connections 2024-04-28T12:40:33.406 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.339+0000 7f48caa00700 1 --2- 172.21.15.84:0/3989547005 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 0x7f48c41092c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:33.407 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.339+0000 7f48caa00700 1 -- 172.21.15.84:0/3989547005 >> 172.21.15.84:0/3989547005 conn(0x7f48c4104470 msgr2=0x7f48c41068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:33.407 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.339+0000 7f48caa00700 1 -- 172.21.15.84:0/3989547005 shutdown_connections 2024-04-28T12:40:33.407 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.339+0000 7f48caa00700 1 -- 172.21.15.84:0/3989547005 wait complete. 2024-04-28T12:40:33.407 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.341+0000 7f48caa00700 1 Processor -- start 2024-04-28T12:40:33.407 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.341+0000 7f48caa00700 1 -- start start 2024-04-28T12:40:33.407 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.341+0000 7f48caa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 0x7f48c419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:33.407 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.341+0000 7f48caa00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f48c410d350 con 0x7f48c4108ea0 2024-04-28T12:40:33.407 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.341+0000 7f48c3e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 0x7f48c419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:33.407 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.342+0000 7f48c3e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 0x7f48c419f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37142/0 (socket says 172.21.15.84:37142) 2024-04-28T12:40:33.407 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.342+0000 7f48c3e00700 1 -- 172.21.15.84:0/1795334088 learned_addr learned my addr 172.21.15.84:0/1795334088 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:33.407 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.342+0000 7f48c3e00700 1 -- 172.21.15.84:0/1795334088 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c419f5c0 con 0x7f48c4108ea0 2024-04-28T12:40:33.407 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.342+0000 7f48c3e00700 1 --2- 172.21.15.84:0/1795334088 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 0x7f48c419f080 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f48b40119f0 tx=0x7f48b4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:33.408 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.342+0000 7f48c0c00700 1 -- 172.21.15.84:0/1795334088 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48b40039e0 con 0x7f48c4108ea0 2024-04-28T12:40:33.408 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.342+0000 7f48c0c00700 1 -- 172.21.15.84:0/1795334088 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48b401b8f0 con 0x7f48c4108ea0 2024-04-28T12:40:33.408 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.342+0000 7f48c0c00700 1 -- 172.21.15.84:0/1795334088 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48b40237a0 con 0x7f48c4108ea0 2024-04-28T12:40:33.408 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.342+0000 7f48caa00700 1 -- 172.21.15.84:0/1795334088 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48c419f850 con 0x7f48c4108ea0 2024-04-28T12:40:33.408 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.343+0000 7f48caa00700 1 -- 172.21.15.84:0/1795334088 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48c41a2540 con 0x7f48c4108ea0 2024-04-28T12:40:33.408 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.344+0000 7f48c0c00700 1 -- 172.21.15.84:0/1795334088 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f48b401b440 con 0x7f48c4108ea0 2024-04-28T12:40:33.408 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.344+0000 7f48c0c00700 1 --2- 172.21.15.84:0/1795334088 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f48a8039610 0x7f48a803bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:33.408 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.344+0000 7f48c0c00700 1 -- 172.21.15.84:0/1795334088 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f48b404e020 con 0x7f48c4108ea0 2024-04-28T12:40:33.408 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.344+0000 7f48caa00700 1 -- 172.21.15.84:0/1795334088 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48c4064a80 con 0x7f48c4108ea0 2024-04-28T12:40:33.408 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.344+0000 7f48c3400700 1 --2- 172.21.15.84:0/1795334088 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f48a8039610 0x7f48a803bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:33.408 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.345+0000 7f48c3400700 1 --2- 172.21.15.84:0/1795334088 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f48a8039610 0x7f48a803bad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f48b0006fd0 tx=0x7f48b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:33.408 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.350+0000 7f48c0c00700 1 -- 172.21.15.84:0/1795334088 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f48b401f030 con 0x7f48c4108ea0 2024-04-28T12:40:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.604+0000 7f48caa00700 1 -- 172.21.15.84:0/1795334088 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f48c4105650 con 0x7f48a8039610 2024-04-28T12:40:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.730+0000 7f48c0c00700 1 -- 172.21.15.84:0/1795334088 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f48c4105650 con 0x7f48a8039610 2024-04-28T12:40:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.733+0000 7f48caa00700 1 -- 172.21.15.84:0/1795334088 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f48a8039610 msgr2=0x7f48a803bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.733+0000 7f48caa00700 1 --2- 172.21.15.84:0/1795334088 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f48a8039610 0x7f48a803bad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f48b0006fd0 tx=0x7f48b0006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.733+0000 7f48caa00700 1 -- 172.21.15.84:0/1795334088 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 msgr2=0x7f48c419f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.733+0000 7f48caa00700 1 --2- 172.21.15.84:0/1795334088 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 0x7f48c419f080 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f48b40119f0 tx=0x7f48b4011ad0 comp rx=0 tx=0).stop 2024-04-28T12:40:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.733+0000 7f48caa00700 1 -- 172.21.15.84:0/1795334088 shutdown_connections 2024-04-28T12:40:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.733+0000 7f48caa00700 1 --2- 172.21.15.84:0/1795334088 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f48a8039610 0x7f48a803bad0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.733+0000 7f48caa00700 1 --2- 172.21.15.84:0/1795334088 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48c4108ea0 0x7f48c419f080 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.733+0000 7f48caa00700 1 -- 172.21.15.84:0/1795334088 >> 172.21.15.84:0/1795334088 conn(0x7f48c4104470 msgr2=0x7f48c4104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.734+0000 7f48caa00700 1 -- 172.21.15.84:0/1795334088 shutdown_connections 2024-04-28T12:40:33.409 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:32.734+0000 7f48caa00700 1 -- 172.21.15.84:0/1795334088 wait complete. 2024-04-28T12:40:33.410 INFO:teuthology.orchestra.run.smithi084.stdout:Creating initial admin user... 2024-04-28T12:40:34.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:33 smithi084 ceph-mon[28081]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:40:34.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:34.003 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:35.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:35 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:35.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:35 smithi084 ceph-mon[28081]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:40:35.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:35 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:35.316 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$eWvMDcmuuW8R5pLc0yreGObif1OVXp4HIXLHcThLMgqDJaxAadlVe", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714308034, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-28T12:40:35.316 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.074+0000 7f393ba00700 1 Processor -- start 2024-04-28T12:40:35.316 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.074+0000 7f393ba00700 1 -- start start 2024-04-28T12:40:35.322 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.074+0000 7f393ba00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 0x7f39341092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:35.322 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.074+0000 7f393ba00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3934109820 con 0x7f3934108ec0 2024-04-28T12:40:35.322 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.074+0000 7f3939000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 0x7f39341092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:35.322 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.075+0000 7f3939000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 0x7f39341092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37156/0 (socket says 172.21.15.84:37156) 2024-04-28T12:40:35.322 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.075+0000 7f3939000700 1 -- 172.21.15.84:0/3422419884 learned_addr learned my addr 172.21.15.84:0/3422419884 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:35.322 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.075+0000 7f3939000700 1 -- 172.21.15.84:0/3422419884 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f393410a060 con 0x7f3934108ec0 2024-04-28T12:40:35.322 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.075+0000 7f3939000700 1 --2- 172.21.15.84:0/3422419884 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 0x7f39341092e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f39280098a0 tx=0x7f3928009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581ffbde57230acf server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:35.323 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.075+0000 7f3933400700 1 -- 172.21.15.84:0/3422419884 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3928004160 con 0x7f3934108ec0 2024-04-28T12:40:35.323 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.076+0000 7f3933400700 1 -- 172.21.15.84:0/3422419884 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3928004300 con 0x7f3934108ec0 2024-04-28T12:40:35.323 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.076+0000 7f3933400700 1 -- 172.21.15.84:0/3422419884 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3928011a20 con 0x7f3934108ec0 2024-04-28T12:40:35.323 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.076+0000 7f393ba00700 1 -- 172.21.15.84:0/3422419884 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 msgr2=0x7f39341092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:35.323 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.076+0000 7f393ba00700 1 --2- 172.21.15.84:0/3422419884 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 0x7f39341092e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f39280098a0 tx=0x7f3928009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:35.323 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.076+0000 7f393ba00700 1 -- 172.21.15.84:0/3422419884 shutdown_connections 2024-04-28T12:40:35.323 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.076+0000 7f393ba00700 1 --2- 172.21.15.84:0/3422419884 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 0x7f39341092e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:35.323 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.076+0000 7f393ba00700 1 -- 172.21.15.84:0/3422419884 >> 172.21.15.84:0/3422419884 conn(0x7f3934104470 msgr2=0x7f39341068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:35.323 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.077+0000 7f393ba00700 1 -- 172.21.15.84:0/3422419884 shutdown_connections 2024-04-28T12:40:35.323 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.077+0000 7f393ba00700 1 -- 172.21.15.84:0/3422419884 wait complete. 2024-04-28T12:40:35.323 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.078+0000 7f393ba00700 1 Processor -- start 2024-04-28T12:40:35.323 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.078+0000 7f393ba00700 1 -- start start 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.078+0000 7f393ba00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 0x7f393419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.078+0000 7f393ba00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f393410d320 con 0x7f3934108ec0 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.078+0000 7f3939000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 0x7f393419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.078+0000 7f3939000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 0x7f393419f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37166/0 (socket says 172.21.15.84:37166) 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.078+0000 7f3939000700 1 -- 172.21.15.84:0/3297427371 learned_addr learned my addr 172.21.15.84:0/3297427371 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.079+0000 7f3939000700 1 -- 172.21.15.84:0/3297427371 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f393419f590 con 0x7f3934108ec0 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.079+0000 7f3939000700 1 --2- 172.21.15.84:0/3297427371 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 0x7f393419f050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3928000c00 tx=0x7f3928011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.079+0000 7f3931600700 1 -- 172.21.15.84:0/3297427371 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3928003aa0 con 0x7f3934108ec0 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.079+0000 7f3931600700 1 -- 172.21.15.84:0/3297427371 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f392801b8f0 con 0x7f3934108ec0 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.079+0000 7f3931600700 1 -- 172.21.15.84:0/3297427371 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3928023890 con 0x7f3934108ec0 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.079+0000 7f393ba00700 1 -- 172.21.15.84:0/3297427371 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f393419f820 con 0x7f3934108ec0 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.080+0000 7f393ba00700 1 -- 172.21.15.84:0/3297427371 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39341a2510 con 0x7f3934108ec0 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.081+0000 7f393ba00700 1 -- 172.21.15.84:0/3297427371 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3934064a80 con 0x7f3934108ec0 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.081+0000 7f3931600700 1 -- 172.21.15.84:0/3297427371 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f392801b440 con 0x7f3934108ec0 2024-04-28T12:40:35.324 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.082+0000 7f3931600700 1 --2- 172.21.15.84:0/3297427371 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f391c039660 0x7f391c03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.082+0000 7f3931600700 1 -- 172.21.15.84:0/3297427371 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f392804e090 con 0x7f3934108ec0 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.082+0000 7f3933e00700 1 --2- 172.21.15.84:0/3297427371 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f391c039660 0x7f391c03bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.083+0000 7f3933e00700 1 --2- 172.21.15.84:0/3297427371 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f391c039660 0x7f391c03bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3920006fd0 tx=0x7f3920006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.087+0000 7f3931600700 1 -- 172.21.15.84:0/3297427371 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f392801f030 con 0x7f3934108ec0 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.337+0000 7f393ba00700 1 -- 172.21.15.84:0/3297427371 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f39341a27a0 con 0x7f391c039660 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.591+0000 7f3931600700 1 -- 172.21.15.84:0/3297427371 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f39341a27a0 con 0x7f391c039660 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.594+0000 7f393ba00700 1 -- 172.21.15.84:0/3297427371 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f391c039660 msgr2=0x7f391c03bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.594+0000 7f393ba00700 1 --2- 172.21.15.84:0/3297427371 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f391c039660 0x7f391c03bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3920006fd0 tx=0x7f3920006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.594+0000 7f393ba00700 1 -- 172.21.15.84:0/3297427371 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 msgr2=0x7f393419f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.594+0000 7f393ba00700 1 --2- 172.21.15.84:0/3297427371 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 0x7f393419f050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3928000c00 tx=0x7f3928011aa0 comp rx=0 tx=0).stop 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.594+0000 7f393ba00700 1 -- 172.21.15.84:0/3297427371 shutdown_connections 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.594+0000 7f393ba00700 1 --2- 172.21.15.84:0/3297427371 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f391c039660 0x7f391c03bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.594+0000 7f393ba00700 1 --2- 172.21.15.84:0/3297427371 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3934108ec0 0x7f393419f050 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.594+0000 7f393ba00700 1 -- 172.21.15.84:0/3297427371 >> 172.21.15.84:0/3297427371 conn(0x7f3934104470 msgr2=0x7f3934104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:35.325 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.595+0000 7f393ba00700 1 -- 172.21.15.84:0/3297427371 shutdown_connections 2024-04-28T12:40:35.326 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:34.595+0000 7f393ba00700 1 -- 172.21.15.84:0/3297427371 wait complete. 2024-04-28T12:40:35.326 INFO:teuthology.orchestra.run.smithi084.stdout:Fetching dashboard port number... 2024-04-28T12:40:37.283 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:37 smithi084 ceph-mon[28081]: mgrmap e13: x(active, since 6s) 2024-04-28T12:40:37.283 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:37 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3594892719' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-28T12:40:37.749 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 8443 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.129+0000 7f2ce1600700 1 Processor -- start 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.129+0000 7f2ce1600700 1 -- start start 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.129+0000 7f2ce1600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 0x7f2cdc07d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.129+0000 7f2ce1600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc07d900 con 0x7f2cdc07eed0 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.130+0000 7f2cdaa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 0x7f2cdc07d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.130+0000 7f2cdaa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 0x7f2cdc07d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37172/0 (socket says 172.21.15.84:37172) 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.130+0000 7f2cdaa00700 1 -- 172.21.15.84:0/2302055449 learned_addr learned my addr 172.21.15.84:0/2302055449 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.130+0000 7f2cdaa00700 1 -- 172.21.15.84:0/2302055449 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc07da80 con 0x7f2cdc07eed0 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.131+0000 7f2cdaa00700 1 --2- 172.21.15.84:0/2302055449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 0x7f2cdc07d330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0098a0 tx=0x7f2ccc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85dc1d8854bf7629 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.131+0000 7f2cd9600700 1 -- 172.21.15.84:0/2302055449 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ccc004160 con 0x7f2cdc07eed0 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.131+0000 7f2cd9600700 1 -- 172.21.15.84:0/2302055449 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ccc004300 con 0x7f2cdc07eed0 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.131+0000 7f2cd9600700 1 -- 172.21.15.84:0/2302055449 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ccc011a20 con 0x7f2cdc07eed0 2024-04-28T12:40:37.750 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.132+0000 7f2ce1600700 1 -- 172.21.15.84:0/2302055449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 msgr2=0x7f2cdc07d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.132+0000 7f2ce1600700 1 --2- 172.21.15.84:0/2302055449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 0x7f2cdc07d330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0098a0 tx=0x7f2ccc009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.132+0000 7f2ce1600700 1 -- 172.21.15.84:0/2302055449 shutdown_connections 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.132+0000 7f2ce1600700 1 --2- 172.21.15.84:0/2302055449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 0x7f2cdc07d330 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.132+0000 7f2ce1600700 1 -- 172.21.15.84:0/2302055449 >> 172.21.15.84:0/2302055449 conn(0x7f2cdc104470 msgr2=0x7f2cdc1068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.132+0000 7f2ce1600700 1 -- 172.21.15.84:0/2302055449 shutdown_connections 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.132+0000 7f2ce1600700 1 -- 172.21.15.84:0/2302055449 wait complete. 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.133+0000 7f2ce1600700 1 Processor -- start 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.133+0000 7f2ce1600700 1 -- start start 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.133+0000 7f2ce1600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 0x7f2cdc19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.133+0000 7f2ce1600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc10f500 con 0x7f2cdc07eed0 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.134+0000 7f2cdaa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 0x7f2cdc19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.134+0000 7f2cdaa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 0x7f2cdc19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37184/0 (socket says 172.21.15.84:37184) 2024-04-28T12:40:37.751 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.134+0000 7f2cdaa00700 1 -- 172.21.15.84:0/3594892719 learned_addr learned my addr 172.21.15.84:0/3594892719 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.134+0000 7f2cdaa00700 1 -- 172.21.15.84:0/3594892719 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc19f5f0 con 0x7f2cdc07eed0 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.134+0000 7f2cdaa00700 1 --2- 172.21.15.84:0/3594892719 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 0x7f2cdc19f0b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0119f0 tx=0x7f2ccc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.135+0000 7f2cd3400700 1 -- 172.21.15.84:0/3594892719 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ccc0039c0 con 0x7f2cdc07eed0 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.135+0000 7f2cd3400700 1 -- 172.21.15.84:0/3594892719 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ccc003b60 con 0x7f2cdc07eed0 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.135+0000 7f2cd3400700 1 -- 172.21.15.84:0/3594892719 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ccc0197a0 con 0x7f2cdc07eed0 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.135+0000 7f2ce1600700 1 -- 172.21.15.84:0/3594892719 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cdc19f880 con 0x7f2cdc07eed0 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.135+0000 7f2ce1600700 1 -- 172.21.15.84:0/3594892719 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cdc1a2570 con 0x7f2cdc07eed0 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.136+0000 7f2ce1600700 1 -- 172.21.15.84:0/3594892719 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cdc064a80 con 0x7f2cdc07eed0 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.136+0000 7f2cd3400700 1 -- 172.21.15.84:0/3594892719 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f2ccc01ab90 con 0x7f2cdc07eed0 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.136+0000 7f2cd3400700 1 --2- 172.21.15.84:0/3594892719 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2cc0039610 0x7f2cc003bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.136+0000 7f2cd3400700 1 -- 172.21.15.84:0/3594892719 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2ccc04de20 con 0x7f2cdc07eed0 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.137+0000 7f2cda000700 1 --2- 172.21.15.84:0/3594892719 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2cc0039610 0x7f2cc003bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.137+0000 7f2cda000700 1 --2- 172.21.15.84:0/3594892719 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2cc0039610 0x7f2cc003bad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8006fd0 tx=0x7f2cc8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:37.752 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.142+0000 7f2cd3400700 1 -- 172.21.15.84:0/3594892719 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2ccc020c80 con 0x7f2cdc07eed0 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.187+0000 7f2cd3400700 1 -- 172.21.15.84:0/3594892719 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2ccc01a7b0 con 0x7f2cdc07eed0 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.372+0000 7f2ce1600700 1 -- 172.21.15.84:0/3594892719 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f2cdc1a2b20 con 0x7f2cdc07eed0 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.373+0000 7f2cd3400700 1 -- 172.21.15.84:0/3594892719 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f2ccc02a2b0 con 0x7f2cdc07eed0 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.376+0000 7f2ce1600700 1 -- 172.21.15.84:0/3594892719 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2cc0039610 msgr2=0x7f2cc003bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.376+0000 7f2ce1600700 1 --2- 172.21.15.84:0/3594892719 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2cc0039610 0x7f2cc003bad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8006fd0 tx=0x7f2cc8006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.376+0000 7f2ce1600700 1 -- 172.21.15.84:0/3594892719 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 msgr2=0x7f2cdc19f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.376+0000 7f2ce1600700 1 --2- 172.21.15.84:0/3594892719 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 0x7f2cdc19f0b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0119f0 tx=0x7f2ccc011ad0 comp rx=0 tx=0).stop 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.376+0000 7f2ce1600700 1 -- 172.21.15.84:0/3594892719 shutdown_connections 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.376+0000 7f2ce1600700 1 --2- 172.21.15.84:0/3594892719 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2cc0039610 0x7f2cc003bad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.376+0000 7f2ce1600700 1 --2- 172.21.15.84:0/3594892719 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2cdc07eed0 0x7f2cdc19f0b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.376+0000 7f2ce1600700 1 -- 172.21.15.84:0/3594892719 >> 172.21.15.84:0/3594892719 conn(0x7f2cdc104470 msgr2=0x7f2cdc105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.376+0000 7f2ce1600700 1 -- 172.21.15.84:0/3594892719 shutdown_connections 2024-04-28T12:40:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:36.376+0000 7f2ce1600700 1 -- 172.21.15.84:0/3594892719 wait complete. 2024-04-28T12:40:37.777 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-28T12:40:37.777 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-04-28T12:40:37.800 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-28T12:40:37.800 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-04-28T12:40:37.800 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-04-28T12:40:37.800 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-28T12:40:37.806 INFO:teuthology.orchestra.run.smithi084.stdout:Ceph Dashboard is now available at: 2024-04-28T12:40:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:40:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: URL: https://smithi084.front.sepia.ceph.com:8443/ 2024-04-28T12:40:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: User: admin 2024-04-28T12:40:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: Password: ms61iqbuu0 2024-04-28T12:40:37.807 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:40:37.808 INFO:teuthology.orchestra.run.smithi084.stdout:Saving cluster configuration to /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config directory 2024-04-28T12:40:37.810 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling autotune for osd_memory_target 2024-04-28T12:40:39.333 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.402+0000 7f30ba800700 1 Processor -- start 2024-04-28T12:40:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.402+0000 7f30ba800700 1 -- start start 2024-04-28T12:40:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.403+0000 7f30ba800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 0x7f30b41120b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.403+0000 7f30ba800700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f30b4112680 con 0x7f30b410fcc0 2024-04-28T12:40:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.403+0000 7f30b3400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 0x7f30b41120b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:39.334 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.403+0000 7f30b3400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 0x7f30b41120b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37190/0 (socket says 172.21.15.84:37190) 2024-04-28T12:40:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.403+0000 7f30b3400700 1 -- 172.21.15.84:0/1110328901 learned_addr learned my addr 172.21.15.84:0/1110328901 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.404+0000 7f30b3400700 1 -- 172.21.15.84:0/1110328901 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b4112800 con 0x7f30b410fcc0 2024-04-28T12:40:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.404+0000 7f30b3400700 1 --2- 172.21.15.84:0/1110328901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 0x7f30b41120b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f30a40098a0 tx=0x7f30a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa4baad1f27393c server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.405+0000 7f30b2000700 1 -- 172.21.15.84:0/1110328901 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30a4004160 con 0x7f30b410fcc0 2024-04-28T12:40:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.405+0000 7f30b2000700 1 -- 172.21.15.84:0/1110328901 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30a4004300 con 0x7f30b410fcc0 2024-04-28T12:40:39.335 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.405+0000 7f30b2000700 1 -- 172.21.15.84:0/1110328901 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30a4011a20 con 0x7f30b410fcc0 2024-04-28T12:40:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.405+0000 7f30ba800700 1 -- 172.21.15.84:0/1110328901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 msgr2=0x7f30b41120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.405+0000 7f30ba800700 1 --2- 172.21.15.84:0/1110328901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 0x7f30b41120b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f30a40098a0 tx=0x7f30a4009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.405+0000 7f30ba800700 1 -- 172.21.15.84:0/1110328901 shutdown_connections 2024-04-28T12:40:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.406+0000 7f30ba800700 1 --2- 172.21.15.84:0/1110328901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 0x7f30b41120b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.406+0000 7f30ba800700 1 -- 172.21.15.84:0/1110328901 >> 172.21.15.84:0/1110328901 conn(0x7f30b407e540 msgr2=0x7f30b4080960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:39.336 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.406+0000 7f30ba800700 1 -- 172.21.15.84:0/1110328901 shutdown_connections 2024-04-28T12:40:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.406+0000 7f30ba800700 1 -- 172.21.15.84:0/1110328901 wait complete. 2024-04-28T12:40:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.407+0000 7f30ba800700 1 Processor -- start 2024-04-28T12:40:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.407+0000 7f30ba800700 1 -- start start 2024-04-28T12:40:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.407+0000 7f30ba800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 0x7f30b41a3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.407+0000 7f30ba800700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f30b41138c0 con 0x7f30b410fcc0 2024-04-28T12:40:39.337 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.407+0000 7f30b3400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 0x7f30b41a3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.407+0000 7f30b3400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 0x7f30b41a3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37200/0 (socket says 172.21.15.84:37200) 2024-04-28T12:40:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.407+0000 7f30b3400700 1 -- 172.21.15.84:0/3015255260 learned_addr learned my addr 172.21.15.84:0/3015255260 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.408+0000 7f30b3400700 1 -- 172.21.15.84:0/3015255260 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b41a38c0 con 0x7f30b410fcc0 2024-04-28T12:40:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.408+0000 7f30b3400700 1 --2- 172.21.15.84:0/3015255260 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 0x7f30b41a3380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f30a4004660 tx=0x7f30a4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.408+0000 7f30abe00700 1 -- 172.21.15.84:0/3015255260 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30a4003aa0 con 0x7f30b410fcc0 2024-04-28T12:40:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.408+0000 7f30abe00700 1 -- 172.21.15.84:0/3015255260 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30a401b8f0 con 0x7f30b410fcc0 2024-04-28T12:40:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.408+0000 7f30ba800700 1 -- 172.21.15.84:0/3015255260 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30b41a3b50 con 0x7f30b410fcc0 2024-04-28T12:40:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.408+0000 7f30abe00700 1 -- 172.21.15.84:0/3015255260 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30a4023900 con 0x7f30b410fcc0 2024-04-28T12:40:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.409+0000 7f30ba800700 1 -- 172.21.15.84:0/3015255260 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30b41a6840 con 0x7f30b410fcc0 2024-04-28T12:40:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.410+0000 7f30ba800700 1 -- 172.21.15.84:0/3015255260 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30b4064a80 con 0x7f30b410fcc0 2024-04-28T12:40:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.410+0000 7f30abe00700 1 -- 172.21.15.84:0/3015255260 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f30a401baa0 con 0x7f30b410fcc0 2024-04-28T12:40:39.339 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.410+0000 7f30abe00700 1 --2- 172.21.15.84:0/3015255260 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3098039700 0x7f309803bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.410+0000 7f30abe00700 1 -- 172.21.15.84:0/3015255260 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f30a404e030 con 0x7f30b410fcc0 2024-04-28T12:40:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.411+0000 7f30b2a00700 1 --2- 172.21.15.84:0/3015255260 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3098039700 0x7f309803bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.411+0000 7f30b2a00700 1 --2- 172.21.15.84:0/3015255260 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3098039700 0x7f309803bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f30a0006fd0 tx=0x7f30a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.415+0000 7f30abe00700 1 -- 172.21.15.84:0/3015255260 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f30a401f030 con 0x7f30b410fcc0 2024-04-28T12:40:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.639+0000 7f30ba800700 1 -- 172.21.15.84:0/3015255260 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f30b41a6f30 con 0x7f30b410fcc0 2024-04-28T12:40:39.340 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.640+0000 7f30abe00700 1 -- 172.21.15.84:0/3015255260 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f30a402bbd0 con 0x7f30b410fcc0 2024-04-28T12:40:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.642+0000 7f30ba800700 1 -- 172.21.15.84:0/3015255260 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3098039700 msgr2=0x7f309803bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.642+0000 7f30ba800700 1 --2- 172.21.15.84:0/3015255260 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3098039700 0x7f309803bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f30a0006fd0 tx=0x7f30a0006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.642+0000 7f30ba800700 1 -- 172.21.15.84:0/3015255260 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 msgr2=0x7f30b41a3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.642+0000 7f30ba800700 1 --2- 172.21.15.84:0/3015255260 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 0x7f30b41a3380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f30a4004660 tx=0x7f30a4011aa0 comp rx=0 tx=0).stop 2024-04-28T12:40:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.643+0000 7f30ba800700 1 -- 172.21.15.84:0/3015255260 shutdown_connections 2024-04-28T12:40:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.643+0000 7f30ba800700 1 --2- 172.21.15.84:0/3015255260 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3098039700 0x7f309803bbc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.643+0000 7f30ba800700 1 --2- 172.21.15.84:0/3015255260 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f30b410fcc0 0x7f30b41a3380 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:39.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.643+0000 7f30ba800700 1 -- 172.21.15.84:0/3015255260 >> 172.21.15.84:0/3015255260 conn(0x7f30b407e540 msgr2=0x7f30b407f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.643+0000 7f30ba800700 1 -- 172.21.15.84:0/3015255260 shutdown_connections 2024-04-28T12:40:39.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:38.643+0000 7f30ba800700 1 -- 172.21.15.84:0/3015255260 wait complete. 2024-04-28T12:40:41.065 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.026+0000 7f1fc0800700 1 Processor -- start 2024-04-28T12:40:41.065 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.026+0000 7f1fc0800700 1 -- start start 2024-04-28T12:40:41.065 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.026+0000 7f1fc0800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 0x7f1fb810b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:41.065 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.026+0000 7f1fc0800700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb810bb50 con 0x7f1fb810b160 2024-04-28T12:40:41.065 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.027+0000 7f1fbde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 0x7f1fb810b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:41.065 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.027+0000 7f1fbde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 0x7f1fb810b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:56578/0 (socket says 172.21.15.84:56578) 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.027+0000 7f1fbde00700 1 -- 172.21.15.84:0/1793312035 learned_addr learned my addr 172.21.15.84:0/1793312035 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.027+0000 7f1fbde00700 1 -- 172.21.15.84:0/1793312035 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb810c310 con 0x7f1fb810b160 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.028+0000 7f1fbde00700 1 --2- 172.21.15.84:0/1793312035 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 0x7f1fb810b580 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0098a0 tx=0x7f1fac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb9c420130fb0fd server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.028+0000 7f1fbca00700 1 -- 172.21.15.84:0/1793312035 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fac004160 con 0x7f1fb810b160 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.028+0000 7f1fbca00700 1 -- 172.21.15.84:0/1793312035 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fac004300 con 0x7f1fb810b160 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.028+0000 7f1fbca00700 1 -- 172.21.15.84:0/1793312035 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fac011a20 con 0x7f1fb810b160 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.029+0000 7f1fc0800700 1 -- 172.21.15.84:0/1793312035 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 msgr2=0x7f1fb810b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.029+0000 7f1fc0800700 1 --2- 172.21.15.84:0/1793312035 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 0x7f1fb810b580 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0098a0 tx=0x7f1fac009bb0 comp rx=0 tx=0).stop 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.029+0000 7f1fc0800700 1 -- 172.21.15.84:0/1793312035 shutdown_connections 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.029+0000 7f1fc0800700 1 --2- 172.21.15.84:0/1793312035 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 0x7f1fb810b580 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.029+0000 7f1fc0800700 1 -- 172.21.15.84:0/1793312035 >> 172.21.15.84:0/1793312035 conn(0x7f1fb807e4e0 msgr2=0x7f1fb807e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.029+0000 7f1fc0800700 1 -- 172.21.15.84:0/1793312035 shutdown_connections 2024-04-28T12:40:41.066 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.029+0000 7f1fc0800700 1 -- 172.21.15.84:0/1793312035 wait complete. 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.030+0000 7f1fc0800700 1 Processor -- start 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.030+0000 7f1fc0800700 1 -- start start 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.030+0000 7f1fc0800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 0x7f1fb819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.030+0000 7f1fc0800700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb810f6c0 con 0x7f1fb810b160 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.031+0000 7f1fbde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 0x7f1fb819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.031+0000 7f1fbde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 0x7f1fb819f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:56592/0 (socket says 172.21.15.84:56592) 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.031+0000 7f1fbde00700 1 -- 172.21.15.84:0/910385314 learned_addr learned my addr 172.21.15.84:0/910385314 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.031+0000 7f1fbde00700 1 -- 172.21.15.84:0/910385314 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb819f5c0 con 0x7f1fb810b160 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.031+0000 7f1fbde00700 1 --2- 172.21.15.84:0/910385314 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 0x7f1fb819f080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1fac000c00 tx=0x7f1fac011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.032+0000 7f1fb2a00700 1 -- 172.21.15.84:0/910385314 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fac003aa0 con 0x7f1fb810b160 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.032+0000 7f1fb2a00700 1 -- 172.21.15.84:0/910385314 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fac01b8f0 con 0x7f1fb810b160 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.032+0000 7f1fc0800700 1 -- 172.21.15.84:0/910385314 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb819f850 con 0x7f1fb810b160 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.032+0000 7f1fb2a00700 1 -- 172.21.15.84:0/910385314 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fac023890 con 0x7f1fb810b160 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.032+0000 7f1fc0800700 1 -- 172.21.15.84:0/910385314 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb81a2540 con 0x7f1fb810b160 2024-04-28T12:40:41.067 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.033+0000 7f1fc0800700 1 -- 172.21.15.84:0/910385314 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fb8064a80 con 0x7f1fb810b160 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.033+0000 7f1fb2a00700 1 -- 172.21.15.84:0/910385314 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1fac01b440 con 0x7f1fb810b160 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.033+0000 7f1fb2a00700 1 --2- 172.21.15.84:0/910385314 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f1fa0039700 0x7f1fa003bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.033+0000 7f1fb2a00700 1 -- 172.21.15.84:0/910385314 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1fac04dff0 con 0x7f1fb810b160 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.034+0000 7f1fbd400700 1 --2- 172.21.15.84:0/910385314 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f1fa0039700 0x7f1fa003bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.035+0000 7f1fbd400700 1 --2- 172.21.15.84:0/910385314 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f1fa0039700 0x7f1fa003bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8006fd0 tx=0x7f1fa8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.039+0000 7f1fb2a00700 1 -- 172.21.15.84:0/910385314 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1fac01f030 con 0x7f1fb810b160 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.388+0000 7f1fc0800700 1 -- 172.21.15.84:0/910385314 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f1fb81a2710 con 0x7f1fb810b160 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.393+0000 7f1fb2a00700 1 -- 172.21.15.84:0/910385314 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f1fac016260 con 0x7f1fb810b160 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.396+0000 7f1fc0800700 1 -- 172.21.15.84:0/910385314 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f1fa0039700 msgr2=0x7f1fa003bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.396+0000 7f1fc0800700 1 --2- 172.21.15.84:0/910385314 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f1fa0039700 0x7f1fa003bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8006fd0 tx=0x7f1fa8006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.396+0000 7f1fc0800700 1 -- 172.21.15.84:0/910385314 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 msgr2=0x7f1fb819f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.396+0000 7f1fc0800700 1 --2- 172.21.15.84:0/910385314 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 0x7f1fb819f080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1fac000c00 tx=0x7f1fac011aa0 comp rx=0 tx=0).stop 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.396+0000 7f1fc0800700 1 -- 172.21.15.84:0/910385314 shutdown_connections 2024-04-28T12:40:41.068 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.396+0000 7f1fc0800700 1 --2- 172.21.15.84:0/910385314 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f1fa0039700 0x7f1fa003bbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.396+0000 7f1fc0800700 1 --2- 172.21.15.84:0/910385314 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1fb810b160 0x7f1fb819f080 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.396+0000 7f1fc0800700 1 -- 172.21.15.84:0/910385314 >> 172.21.15.84:0/910385314 conn(0x7f1fb807e4e0 msgr2=0x7f1fb8108490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.397+0000 7f1fc0800700 1 -- 172.21.15.84:0/910385314 shutdown_connections 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr 2024-04-28T12:40:40.397+0000 7f1fc0800700 1 -- 172.21.15.84:0/910385314 wait complete. 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout:Or, if you are only running a single cluster on this host: 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-28T12:40:41.069 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:40:41.070 INFO:teuthology.orchestra.run.smithi084.stdout: ceph telemetry on 2024-04-28T12:40:41.070 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:40:41.070 INFO:teuthology.orchestra.run.smithi084.stdout:For more information see: 2024-04-28T12:40:41.070 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:40:41.070 INFO:teuthology.orchestra.run.smithi084.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-28T12:40:41.070 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:40:41.070 INFO:teuthology.orchestra.run.smithi084.stdout:Bootstrap complete. 2024-04-28T12:40:41.167 INFO:tasks.cephadm:Fetching config... 2024-04-28T12:40:41.167 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:40:41.168 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-28T12:40:41.208 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-28T12:40:41.208 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:40:41.208 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-28T12:40:41.285 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-28T12:40:41.285 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:40:41.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-28T12:40:41.398 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-28T12:40:41.399 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:40:41.399 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-28T12:40:41.485 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-28T12:40:41.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAjknDkjQ0gVmdgzc9aJj0bv5gInLLZBiuM3xYG/OSQQzQWfNUrO8P3q9aoLvCprZPGnvJoebwrm3Z/gBNVVeZWKReRB60sk2jRPfu0LhoQtIURNrxaQrMNa0Si+d5jHGqT4PSeWgzYeejapVna1FoRCJKr7rfrzlRFCPL7QbXBfckSFJwpD4lCmNKvdvprjEHoe23Jbcmbt4Luyl9buRQXLW8uKe5ZnPf2MuvK4PImPxSLKtNy/aLTcWwi+wv+xGVEeIczB9FxsVNAqUGyiyHejjXu0NS7bVMgNuHyyL0ay1S1rZ5iuaYPUjWXVqIyJhrkitZVWbqoGDFIJ9HJPDZsO26LfHt+KJJhXNlYBCaR+TuSeq4MsaIChnirG5zfpUD0CdD2IbwNO2C0CbP/lhx5ZPv1rfumbmnOokY0zgZXEPvlRetDGbC3Z+41HxtUobUGF7Y9dI+mi1aF6BK1BxGoH2ZHOHoLpVTf7y4nN4/R/b+CqBaXCNA6snsnt+aMac= ceph-ddee604a-055b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-28T12:40:41.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:41 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/910385314' entity='client.admin' 2024-04-28T12:40:41.615 INFO:teuthology.orchestra.run.smithi084.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAjknDkjQ0gVmdgzc9aJj0bv5gInLLZBiuM3xYG/OSQQzQWfNUrO8P3q9aoLvCprZPGnvJoebwrm3Z/gBNVVeZWKReRB60sk2jRPfu0LhoQtIURNrxaQrMNa0Si+d5jHGqT4PSeWgzYeejapVna1FoRCJKr7rfrzlRFCPL7QbXBfckSFJwpD4lCmNKvdvprjEHoe23Jbcmbt4Luyl9buRQXLW8uKe5ZnPf2MuvK4PImPxSLKtNy/aLTcWwi+wv+xGVEeIczB9FxsVNAqUGyiyHejjXu0NS7bVMgNuHyyL0ay1S1rZ5iuaYPUjWXVqIyJhrkitZVWbqoGDFIJ9HJPDZsO26LfHt+KJJhXNlYBCaR+TuSeq4MsaIChnirG5zfpUD0CdD2IbwNO2C0CbP/lhx5ZPv1rfumbmnOokY0zgZXEPvlRetDGbC3Z+41HxtUobUGF7Y9dI+mi1aF6BK1BxGoH2ZHOHoLpVTf7y4nN4/R/b+CqBaXCNA6snsnt+aMac= ceph-ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:40:41.660 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAjknDkjQ0gVmdgzc9aJj0bv5gInLLZBiuM3xYG/OSQQzQWfNUrO8P3q9aoLvCprZPGnvJoebwrm3Z/gBNVVeZWKReRB60sk2jRPfu0LhoQtIURNrxaQrMNa0Si+d5jHGqT4PSeWgzYeejapVna1FoRCJKr7rfrzlRFCPL7QbXBfckSFJwpD4lCmNKvdvprjEHoe23Jbcmbt4Luyl9buRQXLW8uKe5ZnPf2MuvK4PImPxSLKtNy/aLTcWwi+wv+xGVEeIczB9FxsVNAqUGyiyHejjXu0NS7bVMgNuHyyL0ay1S1rZ5iuaYPUjWXVqIyJhrkitZVWbqoGDFIJ9HJPDZsO26LfHt+KJJhXNlYBCaR+TuSeq4MsaIChnirG5zfpUD0CdD2IbwNO2C0CbP/lhx5ZPv1rfumbmnOokY0zgZXEPvlRetDGbC3Z+41HxtUobUGF7Y9dI+mi1aF6BK1BxGoH2ZHOHoLpVTf7y4nN4/R/b+CqBaXCNA6snsnt+aMac= ceph-ddee604a-055b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-28T12:40:41.700 INFO:teuthology.orchestra.run.smithi161.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAjknDkjQ0gVmdgzc9aJj0bv5gInLLZBiuM3xYG/OSQQzQWfNUrO8P3q9aoLvCprZPGnvJoebwrm3Z/gBNVVeZWKReRB60sk2jRPfu0LhoQtIURNrxaQrMNa0Si+d5jHGqT4PSeWgzYeejapVna1FoRCJKr7rfrzlRFCPL7QbXBfckSFJwpD4lCmNKvdvprjEHoe23Jbcmbt4Luyl9buRQXLW8uKe5ZnPf2MuvK4PImPxSLKtNy/aLTcWwi+wv+xGVEeIczB9FxsVNAqUGyiyHejjXu0NS7bVMgNuHyyL0ay1S1rZ5iuaYPUjWXVqIyJhrkitZVWbqoGDFIJ9HJPDZsO26LfHt+KJJhXNlYBCaR+TuSeq4MsaIChnirG5zfpUD0CdD2IbwNO2C0CbP/lhx5ZPv1rfumbmnOokY0zgZXEPvlRetDGbC3Z+41HxtUobUGF7Y9dI+mi1aF6BK1BxGoH2ZHOHoLpVTf7y4nN4/R/b+CqBaXCNA6snsnt+aMac= ceph-ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:40:41.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAjknDkjQ0gVmdgzc9aJj0bv5gInLLZBiuM3xYG/OSQQzQWfNUrO8P3q9aoLvCprZPGnvJoebwrm3Z/gBNVVeZWKReRB60sk2jRPfu0LhoQtIURNrxaQrMNa0Si+d5jHGqT4PSeWgzYeejapVna1FoRCJKr7rfrzlRFCPL7QbXBfckSFJwpD4lCmNKvdvprjEHoe23Jbcmbt4Luyl9buRQXLW8uKe5ZnPf2MuvK4PImPxSLKtNy/aLTcWwi+wv+xGVEeIczB9FxsVNAqUGyiyHejjXu0NS7bVMgNuHyyL0ay1S1rZ5iuaYPUjWXVqIyJhrkitZVWbqoGDFIJ9HJPDZsO26LfHt+KJJhXNlYBCaR+TuSeq4MsaIChnirG5zfpUD0CdD2IbwNO2C0CbP/lhx5ZPv1rfumbmnOokY0zgZXEPvlRetDGbC3Z+41HxtUobUGF7Y9dI+mi1aF6BK1BxGoH2ZHOHoLpVTf7y4nN4/R/b+CqBaXCNA6snsnt+aMac= ceph-ddee604a-055b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-28T12:40:41.753 INFO:teuthology.orchestra.run.smithi179.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAjknDkjQ0gVmdgzc9aJj0bv5gInLLZBiuM3xYG/OSQQzQWfNUrO8P3q9aoLvCprZPGnvJoebwrm3Z/gBNVVeZWKReRB60sk2jRPfu0LhoQtIURNrxaQrMNa0Si+d5jHGqT4PSeWgzYeejapVna1FoRCJKr7rfrzlRFCPL7QbXBfckSFJwpD4lCmNKvdvprjEHoe23Jbcmbt4Luyl9buRQXLW8uKe5ZnPf2MuvK4PImPxSLKtNy/aLTcWwi+wv+xGVEeIczB9FxsVNAqUGyiyHejjXu0NS7bVMgNuHyyL0ay1S1rZ5iuaYPUjWXVqIyJhrkitZVWbqoGDFIJ9HJPDZsO26LfHt+KJJhXNlYBCaR+TuSeq4MsaIChnirG5zfpUD0CdD2IbwNO2C0CbP/lhx5ZPv1rfumbmnOokY0zgZXEPvlRetDGbC3Z+41HxtUobUGF7Y9dI+mi1aF6BK1BxGoH2ZHOHoLpVTf7y4nN4/R/b+CqBaXCNA6snsnt+aMac= ceph-ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:40:41.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-28T12:40:42.160 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:40:44.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.942+0000 7f41a0c00700 1 -- 172.21.15.84:0/4269423686 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f419c1095e0 msgr2=0x7f419c109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:44.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.942+0000 7f41a0c00700 1 --2- 172.21.15.84:0/4269423686 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f419c1095e0 0x7f419c109a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f418c009910 tx=0x7f418c009c20 comp rx=0 tx=0).stop 2024-04-28T12:40:44.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.942+0000 7f41a0c00700 1 -- 172.21.15.84:0/4269423686 shutdown_connections 2024-04-28T12:40:44.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.942+0000 7f41a0c00700 1 --2- 172.21.15.84:0/4269423686 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f419c1095e0 0x7f419c109a00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:44.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.942+0000 7f41a0c00700 1 -- 172.21.15.84:0/4269423686 >> 172.21.15.84:0/4269423686 conn(0x7f419c078440 msgr2=0x7f419c07a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:44.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.942+0000 7f41a0c00700 1 -- 172.21.15.84:0/4269423686 shutdown_connections 2024-04-28T12:40:44.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.942+0000 7f41a0c00700 1 -- 172.21.15.84:0/4269423686 wait complete. 2024-04-28T12:40:44.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.944+0000 7f41a0c00700 1 Processor -- start 2024-04-28T12:40:44.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.944+0000 7f41a0c00700 1 -- start start 2024-04-28T12:40:44.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.944+0000 7f41a0c00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f419c1095e0 0x7f419c19e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:44.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.944+0000 7f41a0c00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f419c10f1b0 con 0x7f419c1095e0 2024-04-28T12:40:44.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.944+0000 7f419b400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f419c1095e0 0x7f419c19e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:44.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.944+0000 7f419b400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f419c1095e0 0x7f419c19e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:56598/0 (socket says 172.21.15.84:56598) 2024-04-28T12:40:44.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.944+0000 7f419b400700 1 -- 172.21.15.84:0/1149929888 learned_addr learned my addr 172.21.15.84:0/1149929888 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:44.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.945+0000 7f419b400700 1 -- 172.21.15.84:0/1149929888 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419c19ee00 con 0x7f419c1095e0 2024-04-28T12:40:44.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.945+0000 7f419b400700 1 --2- 172.21.15.84:0/1149929888 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f419c1095e0 0x7f419c19e8c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f418c005190 tx=0x7f418c004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:44.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.945+0000 7f4198c00700 1 -- 172.21.15.84:0/1149929888 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f418c027070 con 0x7f419c1095e0 2024-04-28T12:40:44.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.945+0000 7f4198c00700 1 -- 172.21.15.84:0/1149929888 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f418c022470 con 0x7f419c1095e0 2024-04-28T12:40:44.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.945+0000 7f41a0c00700 1 -- 172.21.15.84:0/1149929888 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419c19f090 con 0x7f419c1095e0 2024-04-28T12:40:44.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.945+0000 7f4198c00700 1 -- 172.21.15.84:0/1149929888 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f418c00f5f0 con 0x7f419c1095e0 2024-04-28T12:40:44.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.946+0000 7f41a0c00700 1 -- 172.21.15.84:0/1149929888 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f419c1a1d80 con 0x7f419c1095e0 2024-04-28T12:40:44.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.947+0000 7f41a0c00700 1 -- 172.21.15.84:0/1149929888 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f419c10f930 con 0x7f419c1095e0 2024-04-28T12:40:44.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.947+0000 7f4198c00700 1 -- 172.21.15.84:0/1149929888 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f418c02a070 con 0x7f419c1095e0 2024-04-28T12:40:44.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.948+0000 7f4198c00700 1 --2- 172.21.15.84:0/1149929888 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f417c0396b0 0x7f417c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:44.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.948+0000 7f4198c00700 1 -- 172.21.15.84:0/1149929888 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f418c01e030 con 0x7f419c1095e0 2024-04-28T12:40:44.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.948+0000 7f4193e00700 1 --2- 172.21.15.84:0/1149929888 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f417c0396b0 0x7f417c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:44.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.949+0000 7f4193e00700 1 --2- 172.21.15.84:0/1149929888 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f417c0396b0 0x7f417c03bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4184006fd0 tx=0x7f4184006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:44.949 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:44.953+0000 7f4198c00700 1 -- 172.21.15.84:0/1149929888 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f418c013070 con 0x7f419c1095e0 2024-04-28T12:40:45.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:45.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:45.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi084", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:40:45.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:45.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:40:45.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:45.172 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:45.176+0000 7f41a0c00700 1 -- 172.21.15.84:0/1149929888 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f419c02e5a0 con 0x7f419c1095e0 2024-04-28T12:40:45.175 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:45.179+0000 7f4198c00700 1 -- 172.21.15.84:0/1149929888 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f418c030020 con 0x7f419c1095e0 2024-04-28T12:40:45.179 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:45.183+0000 7f41a0c00700 1 -- 172.21.15.84:0/1149929888 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f417c0396b0 msgr2=0x7f417c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:45.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:45.183+0000 7f41a0c00700 1 --2- 172.21.15.84:0/1149929888 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f417c0396b0 0x7f417c03bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4184006fd0 tx=0x7f4184006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:45.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:45.183+0000 7f41a0c00700 1 -- 172.21.15.84:0/1149929888 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f419c1095e0 msgr2=0x7f419c19e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:45.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:45.183+0000 7f41a0c00700 1 --2- 172.21.15.84:0/1149929888 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f419c1095e0 0x7f419c19e8c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f418c005190 tx=0x7f418c004f80 comp rx=0 tx=0).stop 2024-04-28T12:40:45.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:45.183+0000 7f41a0c00700 1 -- 172.21.15.84:0/1149929888 shutdown_connections 2024-04-28T12:40:45.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:45.183+0000 7f41a0c00700 1 --2- 172.21.15.84:0/1149929888 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f417c0396b0 0x7f417c03bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:45.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:45.183+0000 7f41a0c00700 1 --2- 172.21.15.84:0/1149929888 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f419c1095e0 0x7f419c19e8c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:45.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:45.183+0000 7f41a0c00700 1 -- 172.21.15.84:0/1149929888 >> 172.21.15.84:0/1149929888 conn(0x7f419c078440 msgr2=0x7f419c078ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:45.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:45.184+0000 7f41a0c00700 1 -- 172.21.15.84:0/1149929888 shutdown_connections 2024-04-28T12:40:45.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:45.184+0000 7f41a0c00700 1 -- 172.21.15.84:0/1149929888 wait complete. 2024-04-28T12:40:45.932 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-28T12:40:45.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-28T12:40:46.364 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:40:46.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:46 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1149929888' entity='client.admin' 2024-04-28T12:40:46.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:40:46.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:48.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.431+0000 7ffbcd400700 1 -- 172.21.15.84:0/336776204 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbc8077dd0 msgr2=0x7ffbc80781f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:48.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.431+0000 7ffbcd400700 1 --2- 172.21.15.84:0/336776204 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbc8077dd0 0x7ffbc80781f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009910 tx=0x7ffbb8009c20 comp rx=0 tx=0).stop 2024-04-28T12:40:48.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.431+0000 7ffbcd400700 1 -- 172.21.15.84:0/336776204 shutdown_connections 2024-04-28T12:40:48.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.431+0000 7ffbcd400700 1 --2- 172.21.15.84:0/336776204 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbc8077dd0 0x7ffbc80781f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:48.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.431+0000 7ffbcd400700 1 -- 172.21.15.84:0/336776204 >> 172.21.15.84:0/336776204 conn(0x7ffbc81006a0 msgr2=0x7ffbc8102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:48.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.431+0000 7ffbcd400700 1 -- 172.21.15.84:0/336776204 shutdown_connections 2024-04-28T12:40:48.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.431+0000 7ffbcd400700 1 -- 172.21.15.84:0/336776204 wait complete. 2024-04-28T12:40:48.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.432+0000 7ffbcd400700 1 Processor -- start 2024-04-28T12:40:48.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.433+0000 7ffbcd400700 1 -- start start 2024-04-28T12:40:48.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.433+0000 7ffbcd400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbc8077dd0 0x7ffbc819ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:48.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.433+0000 7ffbcd400700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc810f4a0 con 0x7ffbc8077dd0 2024-04-28T12:40:48.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.433+0000 7ffbc6800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbc8077dd0 0x7ffbc819ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:48.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.433+0000 7ffbc6800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbc8077dd0 0x7ffbc819ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:56610/0 (socket says 172.21.15.84:56610) 2024-04-28T12:40:48.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.433+0000 7ffbc6800700 1 -- 172.21.15.84:0/3907092449 learned_addr learned my addr 172.21.15.84:0/3907092449 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:48.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.434+0000 7ffbc6800700 1 -- 172.21.15.84:0/3907092449 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc819efb0 con 0x7ffbc8077dd0 2024-04-28T12:40:48.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.434+0000 7ffbc6800700 1 --2- 172.21.15.84:0/3907092449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbc8077dd0 0x7ffbc819ea70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8005b40 tx=0x7ffbb8004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:48.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.434+0000 7ffbbf400700 1 -- 172.21.15.84:0/3907092449 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbb8027070 con 0x7ffbc8077dd0 2024-04-28T12:40:48.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.435+0000 7ffbbf400700 1 -- 172.21.15.84:0/3907092449 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb800be10 con 0x7ffbc8077dd0 2024-04-28T12:40:48.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.435+0000 7ffbbf400700 1 -- 172.21.15.84:0/3907092449 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbb8022de0 con 0x7ffbc8077dd0 2024-04-28T12:40:48.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.435+0000 7ffbcd400700 1 -- 172.21.15.84:0/3907092449 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc819f240 con 0x7ffbc8077dd0 2024-04-28T12:40:48.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.435+0000 7ffbcd400700 1 -- 172.21.15.84:0/3907092449 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc81a1f30 con 0x7ffbc8077dd0 2024-04-28T12:40:48.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.436+0000 7ffbbf400700 1 -- 172.21.15.84:0/3907092449 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ffbb802a070 con 0x7ffbc8077dd0 2024-04-28T12:40:48.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.436+0000 7ffbcd400700 1 -- 172.21.15.84:0/3907092449 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbc810fb70 con 0x7ffbc8077dd0 2024-04-28T12:40:48.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.436+0000 7ffbbf400700 1 --2- 172.21.15.84:0/3907092449 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffbac0396b0 0x7ffbac03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:48.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.436+0000 7ffbbf400700 1 -- 172.21.15.84:0/3907092449 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ffbb801e030 con 0x7ffbc8077dd0 2024-04-28T12:40:48.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.437+0000 7ffbbfe00700 1 --2- 172.21.15.84:0/3907092449 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffbac0396b0 0x7ffbac03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:48.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.437+0000 7ffbbfe00700 1 --2- 172.21.15.84:0/3907092449 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffbac0396b0 0x7ffbac03bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0006fd0 tx=0x7ffbb0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.442+0000 7ffbbf400700 1 -- 172.21.15.84:0/3907092449 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffbb8013070 con 0x7ffbc8077dd0 2024-04-28T12:40:48.667 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.671+0000 7ffbcd400700 1 -- 172.21.15.84:0/3907092449 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7ffbc802e630 con 0x7ffbac0396b0 2024-04-28T12:40:48.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.679+0000 7ffbbf400700 1 -- 172.21.15.84:0/3907092449 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ffbc802e630 con 0x7ffbac0396b0 2024-04-28T12:40:48.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.682+0000 7ffbcd400700 1 -- 172.21.15.84:0/3907092449 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffbac0396b0 msgr2=0x7ffbac03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:48.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.682+0000 7ffbcd400700 1 --2- 172.21.15.84:0/3907092449 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffbac0396b0 0x7ffbac03bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0006fd0 tx=0x7ffbb0006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:48.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.682+0000 7ffbcd400700 1 -- 172.21.15.84:0/3907092449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbc8077dd0 msgr2=0x7ffbc819ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:48.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.682+0000 7ffbcd400700 1 --2- 172.21.15.84:0/3907092449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbc8077dd0 0x7ffbc819ea70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8005b40 tx=0x7ffbb8004f80 comp rx=0 tx=0).stop 2024-04-28T12:40:48.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.682+0000 7ffbcd400700 1 -- 172.21.15.84:0/3907092449 shutdown_connections 2024-04-28T12:40:48.679 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.682+0000 7ffbcd400700 1 --2- 172.21.15.84:0/3907092449 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffbac0396b0 0x7ffbac03bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:48.679 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.682+0000 7ffbcd400700 1 --2- 172.21.15.84:0/3907092449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbc8077dd0 0x7ffbc819ea70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:48.679 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.682+0000 7ffbcd400700 1 -- 172.21.15.84:0/3907092449 >> 172.21.15.84:0/3907092449 conn(0x7ffbc81006a0 msgr2=0x7ffbc8101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:48.679 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.682+0000 7ffbcd400700 1 -- 172.21.15.84:0/3907092449 shutdown_connections 2024-04-28T12:40:48.679 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:48.682+0000 7ffbcd400700 1 -- 172.21.15.84:0/3907092449 wait complete. 2024-04-28T12:40:49.485 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi161 2024-04-28T12:40:49.485 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-28T12:40:49.485 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/etc/ceph/ceph.conf 2024-04-28T12:40:49.503 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-28T12:40:49.504 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:40:49.560 INFO:tasks.cephadm:Adding host smithi161 to orchestrator... 2024-04-28T12:40:49.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch host add smithi161 2024-04-28T12:40:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:49 smithi084 ceph-mon[28081]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:40:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:40:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:40:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:40:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:49 smithi084 ceph-mon[28081]: Updating smithi084:/etc/ceph/ceph.conf 2024-04-28T12:40:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:49 smithi084 ceph-mon[28081]: Updating smithi084:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:40:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:49 smithi084 ceph-mon[28081]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:40:49.989 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:40:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:50 smithi084 ceph-mon[28081]: Updating smithi084:/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:40:52.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.224+0000 7f2b60000700 1 -- 172.21.15.84:0/407198028 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b5810e3d0 msgr2=0x7f2b5810e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:52.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.224+0000 7f2b60000700 1 --2- 172.21.15.84:0/407198028 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b5810e3d0 0x7f2b5810e7b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c009910 tx=0x7f2b4c009c20 comp rx=0 tx=0).stop 2024-04-28T12:40:52.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.224+0000 7f2b60000700 1 -- 172.21.15.84:0/407198028 shutdown_connections 2024-04-28T12:40:52.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.224+0000 7f2b60000700 1 --2- 172.21.15.84:0/407198028 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b5810e3d0 0x7f2b5810e7b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:52.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.224+0000 7f2b60000700 1 -- 172.21.15.84:0/407198028 >> 172.21.15.84:0/407198028 conn(0x7f2b58078bc0 msgr2=0x7f2b5807afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:52.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.224+0000 7f2b60000700 1 -- 172.21.15.84:0/407198028 shutdown_connections 2024-04-28T12:40:52.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.224+0000 7f2b60000700 1 -- 172.21.15.84:0/407198028 wait complete. 2024-04-28T12:40:52.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.225+0000 7f2b60000700 1 Processor -- start 2024-04-28T12:40:52.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.225+0000 7f2b60000700 1 -- start start 2024-04-28T12:40:52.223 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.226+0000 7f2b60000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b5810e3d0 0x7f2b5819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:52.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.226+0000 7f2b60000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2b58113940 con 0x7f2b5810e3d0 2024-04-28T12:40:52.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.226+0000 7f2b5d600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b5810e3d0 0x7f2b5819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:52.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.226+0000 7f2b5d600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b5810e3d0 0x7f2b5819eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54828/0 (socket says 172.21.15.84:54828) 2024-04-28T12:40:52.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.226+0000 7f2b5d600700 1 -- 172.21.15.84:0/738034867 learned_addr learned my addr 172.21.15.84:0/738034867 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:40:52.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.227+0000 7f2b5d600700 1 -- 172.21.15.84:0/738034867 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b5819f420 con 0x7f2b5810e3d0 2024-04-28T12:40:52.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.227+0000 7f2b5d600700 1 --2- 172.21.15.84:0/738034867 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b5810e3d0 0x7f2b5819eee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c006010 tx=0x7f2b4c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:52.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.227+0000 7f2b56000700 1 -- 172.21.15.84:0/738034867 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b4c026070 con 0x7f2b5810e3d0 2024-04-28T12:40:52.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.227+0000 7f2b56000700 1 -- 172.21.15.84:0/738034867 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b4c00bba0 con 0x7f2b5810e3d0 2024-04-28T12:40:52.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.227+0000 7f2b56000700 1 -- 172.21.15.84:0/738034867 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b4c003930 con 0x7f2b5810e3d0 2024-04-28T12:40:52.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.227+0000 7f2b60000700 1 -- 172.21.15.84:0/738034867 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b5819f710 con 0x7f2b5810e3d0 2024-04-28T12:40:52.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.227+0000 7f2b60000700 1 -- 172.21.15.84:0/738034867 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b581a37b0 con 0x7f2b5810e3d0 2024-04-28T12:40:52.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.228+0000 7f2b60000700 1 -- 172.21.15.84:0/738034867 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b5810ab60 con 0x7f2b5810e3d0 2024-04-28T12:40:52.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.229+0000 7f2b56000700 1 -- 172.21.15.84:0/738034867 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2b4c029070 con 0x7f2b5810e3d0 2024-04-28T12:40:52.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.229+0000 7f2b56000700 1 --2- 172.21.15.84:0/738034867 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2b400396b0 0x7f2b4003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:40:52.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.229+0000 7f2b56000700 1 -- 172.21.15.84:0/738034867 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2b4c01e070 con 0x7f2b5810e3d0 2024-04-28T12:40:52.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.229+0000 7f2b5cc00700 1 --2- 172.21.15.84:0/738034867 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2b400396b0 0x7f2b4003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:40:52.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.230+0000 7f2b5cc00700 1 --2- 172.21.15.84:0/738034867 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2b400396b0 0x7f2b4003bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2b44006fd0 tx=0x7f2b44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:40:52.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.235+0000 7f2b56000700 1 -- 172.21.15.84:0/738034867 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2b4c013070 con 0x7f2b5810e3d0 2024-04-28T12:40:52.461 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:52.465+0000 7f2b60000700 1 -- 172.21.15.84:0/738034867 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi161", "target": ["mon-mgr", ""]}) -- 0x7f2b581a0240 con 0x7f2b400396b0 2024-04-28T12:40:52.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:52 smithi084 ceph-mon[28081]: Updating smithi084:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T12:40:52.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:52.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:52.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:52.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:52 smithi084 ceph-mon[28081]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:40:53.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:53 smithi084 ceph-mon[28081]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi161", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:40:54.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:54 smithi084 ceph-mon[28081]: Deploying cephadm binary to smithi161 2024-04-28T12:40:54.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:54 smithi084 ceph-mon[28081]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:40:56.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:56 smithi084 ceph-mon[28081]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:40:57.237 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:57.240+0000 7f2b56000700 1 -- 172.21.15.84:0/738034867 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f2b581a0240 con 0x7f2b400396b0 2024-04-28T12:40:57.237 INFO:teuthology.orchestra.run.smithi084.stdout:Added host 'smithi161' with addr '172.21.15.161' 2024-04-28T12:40:57.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:57.245+0000 7f2b60000700 1 -- 172.21.15.84:0/738034867 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2b400396b0 msgr2=0x7f2b4003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:57.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:57.245+0000 7f2b60000700 1 --2- 172.21.15.84:0/738034867 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2b400396b0 0x7f2b4003bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2b44006fd0 tx=0x7f2b44006e40 comp rx=0 tx=0).stop 2024-04-28T12:40:57.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:57.245+0000 7f2b60000700 1 -- 172.21.15.84:0/738034867 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b5810e3d0 msgr2=0x7f2b5819eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:40:57.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:57.245+0000 7f2b60000700 1 --2- 172.21.15.84:0/738034867 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b5810e3d0 0x7f2b5819eee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c006010 tx=0x7f2b4c005780 comp rx=0 tx=0).stop 2024-04-28T12:40:57.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:57.245+0000 7f2b60000700 1 -- 172.21.15.84:0/738034867 shutdown_connections 2024-04-28T12:40:57.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:57.245+0000 7f2b60000700 1 --2- 172.21.15.84:0/738034867 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2b400396b0 0x7f2b4003bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:57.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:57.245+0000 7f2b60000700 1 --2- 172.21.15.84:0/738034867 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b5810e3d0 0x7f2b5819eee0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:40:57.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:57.245+0000 7f2b60000700 1 -- 172.21.15.84:0/738034867 >> 172.21.15.84:0/738034867 conn(0x7f2b58078bc0 msgr2=0x7f2b58110f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:40:57.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:57.245+0000 7f2b60000700 1 -- 172.21.15.84:0/738034867 shutdown_connections 2024-04-28T12:40:57.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:40:57.245+0000 7f2b60000700 1 -- 172.21.15.84:0/738034867 wait complete. 2024-04-28T12:40:58.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-28T12:40:58.473 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:40:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:58 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:58 smithi084 ceph-mon[28081]: Added host smithi161 2024-04-28T12:40:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:58 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:40:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:58 smithi084 ceph-mon[28081]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:40:58.620 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:58 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:40:59.639 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:40:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:00.468 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.472+0000 7f8df0000700 1 -- 172.21.15.84:0/1075546700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8de8077950 msgr2=0x7f8de8077d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:00.468 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.472+0000 7f8df0000700 1 --2- 172.21.15.84:0/1075546700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8de8077950 0x7f8de8077d30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc009910 tx=0x7f8ddc009c20 comp rx=0 tx=0).stop 2024-04-28T12:41:00.469 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.472+0000 7f8df0000700 1 -- 172.21.15.84:0/1075546700 shutdown_connections 2024-04-28T12:41:00.469 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.472+0000 7f8df0000700 1 --2- 172.21.15.84:0/1075546700 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8de8077950 0x7f8de8077d30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:00.469 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.472+0000 7f8df0000700 1 -- 172.21.15.84:0/1075546700 >> 172.21.15.84:0/1075546700 conn(0x7f8de8100b30 msgr2=0x7f8de8102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:00.469 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.472+0000 7f8df0000700 1 -- 172.21.15.84:0/1075546700 shutdown_connections 2024-04-28T12:41:00.469 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.472+0000 7f8df0000700 1 -- 172.21.15.84:0/1075546700 wait complete. 2024-04-28T12:41:00.469 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.473+0000 7f8df0000700 1 Processor -- start 2024-04-28T12:41:00.470 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.474+0000 7f8df0000700 1 -- start start 2024-04-28T12:41:00.470 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.474+0000 7f8df0000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8de8077950 0x7f8de819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:00.470 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.474+0000 7f8df0000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8de8113960 con 0x7f8de8077950 2024-04-28T12:41:00.470 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.474+0000 7f8ded600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8de8077950 0x7f8de819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:00.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.474+0000 7f8ded600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8de8077950 0x7f8de819ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:57594/0 (socket says 172.21.15.84:57594) 2024-04-28T12:41:00.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.474+0000 7f8ded600700 1 -- 172.21.15.84:0/2219852611 learned_addr learned my addr 172.21.15.84:0/2219852611 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:41:00.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.475+0000 7f8ded600700 1 -- 172.21.15.84:0/2219852611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de819f3d0 con 0x7f8de8077950 2024-04-28T12:41:00.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.475+0000 7f8ded600700 1 --2- 172.21.15.84:0/2219852611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8de8077950 0x7f8de819ee90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0056d0 tx=0x7f8ddc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:00.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.475+0000 7f8de6000700 1 -- 172.21.15.84:0/2219852611 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ddc026070 con 0x7f8de8077950 2024-04-28T12:41:00.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.475+0000 7f8de6000700 1 -- 172.21.15.84:0/2219852611 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ddc00bc00 con 0x7f8de8077950 2024-04-28T12:41:00.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.475+0000 7f8df0000700 1 -- 172.21.15.84:0/2219852611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de819f660 con 0x7f8de8077950 2024-04-28T12:41:00.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.475+0000 7f8de6000700 1 -- 172.21.15.84:0/2219852611 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ddc003920 con 0x7f8de8077950 2024-04-28T12:41:00.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.475+0000 7f8df0000700 1 -- 172.21.15.84:0/2219852611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de819fac0 con 0x7f8de8077950 2024-04-28T12:41:00.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.477+0000 7f8de6000700 1 -- 172.21.15.84:0/2219852611 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8ddc029070 con 0x7f8de8077950 2024-04-28T12:41:00.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.477+0000 7f8df0000700 1 -- 172.21.15.84:0/2219852611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de8079090 con 0x7f8de8077950 2024-04-28T12:41:00.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.477+0000 7f8de6000700 1 --2- 172.21.15.84:0/2219852611 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8dd0039650 0x7f8dd003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:00.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.477+0000 7f8de6000700 1 -- 172.21.15.84:0/2219852611 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8ddc01e070 con 0x7f8de8077950 2024-04-28T12:41:00.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.477+0000 7f8decc00700 1 --2- 172.21.15.84:0/2219852611 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8dd0039650 0x7f8dd003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:00.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.478+0000 7f8decc00700 1 --2- 172.21.15.84:0/2219852611 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8dd0039650 0x7f8dd003bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4006fd0 tx=0x7f8dd4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:00.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.483+0000 7f8de6000700 1 -- 172.21.15.84:0/2219852611 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8ddc013070 con 0x7f8de8077950 2024-04-28T12:41:00.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:00 smithi084 ceph-mon[28081]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:00.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:00 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:00.714 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.718+0000 7f8df0000700 1 -- 172.21.15.84:0/2219852611 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8de81a01a0 con 0x7f8dd0039650 2024-04-28T12:41:00.715 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.720+0000 7f8de6000700 1 -- 172.21.15.84:0/2219852611 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f8de81a01a0 con 0x7f8dd0039650 2024-04-28T12:41:00.716 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:41:00.716 INFO:teuthology.orchestra.run.smithi084.stdout:[{"addr": "172.21.15.84", "hostname": "smithi084", "labels": [], "status": ""}, {"addr": "172.21.15.161", "hostname": "smithi161", "labels": [], "status": ""}] 2024-04-28T12:41:00.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.722+0000 7f8df0000700 1 -- 172.21.15.84:0/2219852611 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8dd0039650 msgr2=0x7f8dd003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:00.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.722+0000 7f8df0000700 1 --2- 172.21.15.84:0/2219852611 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8dd0039650 0x7f8dd003bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4006fd0 tx=0x7f8dd4006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:00.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.722+0000 7f8df0000700 1 -- 172.21.15.84:0/2219852611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8de8077950 msgr2=0x7f8de819ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:00.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.722+0000 7f8df0000700 1 --2- 172.21.15.84:0/2219852611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8de8077950 0x7f8de819ee90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0056d0 tx=0x7f8ddc0057b0 comp rx=0 tx=0).stop 2024-04-28T12:41:00.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.722+0000 7f8df0000700 1 -- 172.21.15.84:0/2219852611 shutdown_connections 2024-04-28T12:41:00.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.722+0000 7f8df0000700 1 --2- 172.21.15.84:0/2219852611 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8dd0039650 0x7f8dd003bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:00.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.722+0000 7f8df0000700 1 --2- 172.21.15.84:0/2219852611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8de8077950 0x7f8de819ee90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:00.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.723+0000 7f8df0000700 1 -- 172.21.15.84:0/2219852611 >> 172.21.15.84:0/2219852611 conn(0x7f8de8100b30 msgr2=0x7f8de8100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:00.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.723+0000 7f8df0000700 1 -- 172.21.15.84:0/2219852611 shutdown_connections 2024-04-28T12:41:00.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:00.723+0000 7f8df0000700 1 -- 172.21.15.84:0/2219852611 wait complete. 2024-04-28T12:41:01.494 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi179 2024-04-28T12:41:01.494 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-28T12:41:01.494 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/etc/ceph/ceph.conf 2024-04-28T12:41:01.513 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-28T12:41:01.513 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:01.571 INFO:tasks.cephadm:Adding host smithi179 to orchestrator... 2024-04-28T12:41:01.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch host add smithi179 2024-04-28T12:41:01.995 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:41:02.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:02 smithi084 ceph-mon[28081]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:41:02.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:02 smithi084 ceph-mon[28081]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:04.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.224+0000 7fbf02800700 1 -- 172.21.15.84:0/4248533617 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbefc1052b0 msgr2=0x7fbefc105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:04.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.224+0000 7fbf02800700 1 --2- 172.21.15.84:0/4248533617 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbefc1052b0 0x7fbefc105690 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbeec009910 tx=0x7fbeec009c20 comp rx=0 tx=0).stop 2024-04-28T12:41:04.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.225+0000 7fbf02800700 1 -- 172.21.15.84:0/4248533617 shutdown_connections 2024-04-28T12:41:04.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.225+0000 7fbf02800700 1 --2- 172.21.15.84:0/4248533617 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbefc1052b0 0x7fbefc105690 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:04.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.225+0000 7fbf02800700 1 -- 172.21.15.84:0/4248533617 >> 172.21.15.84:0/4248533617 conn(0x7fbefc100b40 msgr2=0x7fbefc102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:04.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.225+0000 7fbf02800700 1 -- 172.21.15.84:0/4248533617 shutdown_connections 2024-04-28T12:41:04.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.225+0000 7fbf02800700 1 -- 172.21.15.84:0/4248533617 wait complete. 2024-04-28T12:41:04.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:04 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:04.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:04 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:04.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:04 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:04.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:04 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:04.222 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:04 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:41:04.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:04 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:04.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:04 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:04.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:04 smithi084 ceph-mon[28081]: Updating smithi161:/etc/ceph/ceph.conf 2024-04-28T12:41:04.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:04 smithi084 ceph-mon[28081]: Updating smithi161:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:04.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:04 smithi084 ceph-mon[28081]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:04.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.226+0000 7fbf02800700 1 Processor -- start 2024-04-28T12:41:04.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.226+0000 7fbf02800700 1 -- start start 2024-04-28T12:41:04.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.226+0000 7fbf02800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbefc1052b0 0x7fbefc1a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:04.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.226+0000 7fbf02800700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbefc10f530 con 0x7fbefc1052b0 2024-04-28T12:41:04.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.227+0000 7fbefb400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbefc1052b0 0x7fbefc1a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:04.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.227+0000 7fbefb400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbefc1052b0 0x7fbefc1a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:57614/0 (socket says 172.21.15.84:57614) 2024-04-28T12:41:04.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.227+0000 7fbefb400700 1 -- 172.21.15.84:0/564571157 learned_addr learned my addr 172.21.15.84:0/564571157 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:41:04.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.227+0000 7fbefb400700 1 -- 172.21.15.84:0/564571157 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbefc1a0c50 con 0x7fbefc1052b0 2024-04-28T12:41:04.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.227+0000 7fbefb400700 1 --2- 172.21.15.84:0/564571157 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbefc1052b0 0x7fbefc1a0710 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbeec006010 tx=0x7fbeec005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:04.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.228+0000 7fbef3e00700 1 -- 172.21.15.84:0/564571157 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbeec026070 con 0x7fbefc1052b0 2024-04-28T12:41:04.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.228+0000 7fbf02800700 1 -- 172.21.15.84:0/564571157 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbefc1a0f40 con 0x7fbefc1052b0 2024-04-28T12:41:04.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.228+0000 7fbef3e00700 1 -- 172.21.15.84:0/564571157 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeec00bba0 con 0x7fbefc1052b0 2024-04-28T12:41:04.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.228+0000 7fbef3e00700 1 -- 172.21.15.84:0/564571157 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbeec003910 con 0x7fbefc1052b0 2024-04-28T12:41:04.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.228+0000 7fbf02800700 1 -- 172.21.15.84:0/564571157 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbefc19a8d0 con 0x7fbefc1052b0 2024-04-28T12:41:04.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.229+0000 7fbf02800700 1 -- 172.21.15.84:0/564571157 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbefc106790 con 0x7fbefc1052b0 2024-04-28T12:41:04.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.229+0000 7fbef3e00700 1 -- 172.21.15.84:0/564571157 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fbeec029070 con 0x7fbefc1052b0 2024-04-28T12:41:04.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.230+0000 7fbef3e00700 1 --2- 172.21.15.84:0/564571157 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbee0039650 0x7fbee003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:04.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.230+0000 7fbef3e00700 1 -- 172.21.15.84:0/564571157 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbeec01e070 con 0x7fbefc1052b0 2024-04-28T12:41:04.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.230+0000 7fbefaa00700 1 --2- 172.21.15.84:0/564571157 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbee0039650 0x7fbee003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:04.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.231+0000 7fbefaa00700 1 --2- 172.21.15.84:0/564571157 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbee0039650 0x7fbee003bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbee8006fd0 tx=0x7fbee8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:04.231 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.235+0000 7fbef3e00700 1 -- 172.21.15.84:0/564571157 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbeec013070 con 0x7fbefc1052b0 2024-04-28T12:41:04.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:04.467+0000 7fbf02800700 1 -- 172.21.15.84:0/564571157 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}) -- 0x7fbefc19be00 con 0x7fbee0039650 2024-04-28T12:41:05.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:05 smithi084 ceph-mon[28081]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:05.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:05 smithi084 ceph-mon[28081]: Updating smithi161:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T12:41:05.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:05 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:05.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:05 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:05.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:05 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:05.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:05 smithi084 ceph-mon[28081]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:06.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:06 smithi084 ceph-mon[28081]: Deploying cephadm binary to smithi179 2024-04-28T12:41:06.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:06 smithi084 ceph-mon[28081]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:08 smithi084 ceph-mon[28081]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:09.179 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:09.181+0000 7fbef3e00700 1 -- 172.21.15.84:0/564571157 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fbefc19be00 con 0x7fbee0039650 2024-04-28T12:41:09.179 INFO:teuthology.orchestra.run.smithi084.stdout:Added host 'smithi179' with addr '172.21.15.179' 2024-04-28T12:41:09.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:09.185+0000 7fbf02800700 1 -- 172.21.15.84:0/564571157 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbee0039650 msgr2=0x7fbee003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:09.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:09.185+0000 7fbf02800700 1 --2- 172.21.15.84:0/564571157 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbee0039650 0x7fbee003bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbee8006fd0 tx=0x7fbee8006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:09.183 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:09.185+0000 7fbf02800700 1 -- 172.21.15.84:0/564571157 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbefc1052b0 msgr2=0x7fbefc1a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:09.183 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:09.185+0000 7fbf02800700 1 --2- 172.21.15.84:0/564571157 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbefc1052b0 0x7fbefc1a0710 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbeec006010 tx=0x7fbeec005780 comp rx=0 tx=0).stop 2024-04-28T12:41:09.183 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:09.185+0000 7fbf02800700 1 -- 172.21.15.84:0/564571157 shutdown_connections 2024-04-28T12:41:09.183 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:09.185+0000 7fbf02800700 1 --2- 172.21.15.84:0/564571157 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbee0039650 0x7fbee003bb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:09.183 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:09.185+0000 7fbf02800700 1 --2- 172.21.15.84:0/564571157 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbefc1052b0 0x7fbefc1a0710 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:09.183 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:09.185+0000 7fbf02800700 1 -- 172.21.15.84:0/564571157 >> 172.21.15.84:0/564571157 conn(0x7fbefc100b40 msgr2=0x7fbefc108050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:09.183 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:09.186+0000 7fbf02800700 1 -- 172.21.15.84:0/564571157 shutdown_connections 2024-04-28T12:41:09.183 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:09.186+0000 7fbf02800700 1 -- 172.21.15.84:0/564571157 wait complete. 2024-04-28T12:41:10.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-28T12:41:10.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:10.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:10 smithi084 ceph-mon[28081]: Added host smithi179 2024-04-28T12:41:10.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:10.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:10 smithi084 ceph-mon[28081]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:10.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:10.446 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:41:11.664 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:12.644 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.647+0000 7f388b200700 1 -- 172.21.15.84:0/1851425689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3884104050 msgr2=0x7f3884104430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:12.645 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.647+0000 7f388b200700 1 --2- 172.21.15.84:0/1851425689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3884104050 0x7f3884104430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3874009910 tx=0x7f3874009c20 comp rx=0 tx=0).stop 2024-04-28T12:41:12.645 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.647+0000 7f388b200700 1 -- 172.21.15.84:0/1851425689 shutdown_connections 2024-04-28T12:41:12.645 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.647+0000 7f388b200700 1 --2- 172.21.15.84:0/1851425689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3884104050 0x7f3884104430 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:12.645 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.647+0000 7f388b200700 1 -- 172.21.15.84:0/1851425689 >> 172.21.15.84:0/1851425689 conn(0x7f38840fd820 msgr2=0x7f38840ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:12.645 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.647+0000 7f388b200700 1 -- 172.21.15.84:0/1851425689 shutdown_connections 2024-04-28T12:41:12.645 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.647+0000 7f388b200700 1 -- 172.21.15.84:0/1851425689 wait complete. 2024-04-28T12:41:12.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:12 smithi084 ceph-mon[28081]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:12.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:12.646 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.648+0000 7f388b200700 1 Processor -- start 2024-04-28T12:41:12.646 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.648+0000 7f388b200700 1 -- start start 2024-04-28T12:41:12.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.649+0000 7f388b200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3884104050 0x7f38841a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:12.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.649+0000 7f388b200700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f388410d380 con 0x7f3884104050 2024-04-28T12:41:12.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.649+0000 7f3883e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3884104050 0x7f38841a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:12.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.649+0000 7f3883e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3884104050 0x7f38841a0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:38896/0 (socket says 172.21.15.84:38896) 2024-04-28T12:41:12.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.649+0000 7f3883e00700 1 -- 172.21.15.84:0/3740943069 learned_addr learned my addr 172.21.15.84:0/3740943069 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:41:12.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.650+0000 7f3883e00700 1 -- 172.21.15.84:0/3740943069 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38841a0c80 con 0x7f3884104050 2024-04-28T12:41:12.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.650+0000 7f3883e00700 1 --2- 172.21.15.84:0/3740943069 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3884104050 0x7f38841a0740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3874006010 tx=0x7f3874005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:12.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.650+0000 7f3880c00700 1 -- 172.21.15.84:0/3740943069 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3874026070 con 0x7f3884104050 2024-04-28T12:41:12.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.650+0000 7f3880c00700 1 -- 172.21.15.84:0/3740943069 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f387400bc00 con 0x7f3884104050 2024-04-28T12:41:12.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.650+0000 7f388b200700 1 -- 172.21.15.84:0/3740943069 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38841a0f10 con 0x7f3884104050 2024-04-28T12:41:12.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.650+0000 7f3880c00700 1 -- 172.21.15.84:0/3740943069 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3874003910 con 0x7f3884104050 2024-04-28T12:41:12.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.651+0000 7f388b200700 1 -- 172.21.15.84:0/3740943069 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f388419a900 con 0x7f3884104050 2024-04-28T12:41:12.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.652+0000 7f388b200700 1 -- 172.21.15.84:0/3740943069 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f388406bab0 con 0x7f3884104050 2024-04-28T12:41:12.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.652+0000 7f3880c00700 1 -- 172.21.15.84:0/3740943069 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3874029070 con 0x7f3884104050 2024-04-28T12:41:12.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.653+0000 7f3880c00700 1 --2- 172.21.15.84:0/3740943069 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f38680396b0 0x7f386803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:12.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.653+0000 7f3880c00700 1 -- 172.21.15.84:0/3740943069 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f387401e070 con 0x7f3884104050 2024-04-28T12:41:12.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.653+0000 7f3883400700 1 --2- 172.21.15.84:0/3740943069 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f38680396b0 0x7f386803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:12.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.654+0000 7f3883400700 1 --2- 172.21.15.84:0/3740943069 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f38680396b0 0x7f386803bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3870006fd0 tx=0x7f3870006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:12.656 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.659+0000 7f3880c00700 1 -- 172.21.15.84:0/3740943069 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3874013070 con 0x7f3884104050 2024-04-28T12:41:12.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.891+0000 7f388b200700 1 -- 172.21.15.84:0/3740943069 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f388419be30 con 0x7f38680396b0 2024-04-28T12:41:12.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.893+0000 7f3880c00700 1 -- 172.21.15.84:0/3740943069 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f388419be30 con 0x7f38680396b0 2024-04-28T12:41:12.891 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:41:12.891 INFO:teuthology.orchestra.run.smithi084.stdout:[{"addr": "172.21.15.84", "hostname": "smithi084", "labels": [], "status": ""}, {"addr": "172.21.15.161", "hostname": "smithi161", "labels": [], "status": ""}, {"addr": "172.21.15.179", "hostname": "smithi179", "labels": [], "status": ""}] 2024-04-28T12:41:12.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.896+0000 7f388b200700 1 -- 172.21.15.84:0/3740943069 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f38680396b0 msgr2=0x7f386803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:12.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.896+0000 7f388b200700 1 --2- 172.21.15.84:0/3740943069 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f38680396b0 0x7f386803bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3870006fd0 tx=0x7f3870006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:12.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.896+0000 7f388b200700 1 -- 172.21.15.84:0/3740943069 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3884104050 msgr2=0x7f38841a0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:12.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.896+0000 7f388b200700 1 --2- 172.21.15.84:0/3740943069 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3884104050 0x7f38841a0740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3874006010 tx=0x7f3874005780 comp rx=0 tx=0).stop 2024-04-28T12:41:12.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.896+0000 7f388b200700 1 -- 172.21.15.84:0/3740943069 shutdown_connections 2024-04-28T12:41:12.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.896+0000 7f388b200700 1 --2- 172.21.15.84:0/3740943069 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f38680396b0 0x7f386803bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:12.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.896+0000 7f388b200700 1 --2- 172.21.15.84:0/3740943069 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3884104050 0x7f38841a0740 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:12.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.896+0000 7f388b200700 1 -- 172.21.15.84:0/3740943069 >> 172.21.15.84:0/3740943069 conn(0x7f38840fd820 msgr2=0x7f3884109210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:12.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.896+0000 7f388b200700 1 -- 172.21.15.84:0/3740943069 shutdown_connections 2024-04-28T12:41:12.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:12.896+0000 7f388b200700 1 -- 172.21.15.84:0/3740943069 wait complete. 2024-04-28T12:41:13.606 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-28T12:41:13.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-28T12:41:14.028 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:41:14.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:14 smithi084 ceph-mon[28081]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:41:14.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:14 smithi084 ceph-mon[28081]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:16.097 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.100+0000 7f110d400700 1 -- 172.21.15.84:0/3678736382 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1108103090 msgr2=0x7f1108103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:16.097 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.100+0000 7f110d400700 1 --2- 172.21.15.84:0/3678736382 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1108103090 0x7f1108103470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009910 tx=0x7f10f8009c20 comp rx=0 tx=0).stop 2024-04-28T12:41:16.098 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.100+0000 7f110d400700 1 -- 172.21.15.84:0/3678736382 shutdown_connections 2024-04-28T12:41:16.098 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.100+0000 7f110d400700 1 --2- 172.21.15.84:0/3678736382 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1108103090 0x7f1108103470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:16.098 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.100+0000 7f110d400700 1 -- 172.21.15.84:0/3678736382 >> 172.21.15.84:0/3678736382 conn(0x7f11080fe960 msgr2=0x7f1108100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:16.098 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.100+0000 7f110d400700 1 -- 172.21.15.84:0/3678736382 shutdown_connections 2024-04-28T12:41:16.098 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.100+0000 7f110d400700 1 -- 172.21.15.84:0/3678736382 wait complete. 2024-04-28T12:41:16.098 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.102+0000 7f110d400700 1 Processor -- start 2024-04-28T12:41:16.099 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.102+0000 7f110d400700 1 -- start start 2024-04-28T12:41:16.099 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.102+0000 7f110d400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1108103090 0x7f11081a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:16.099 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.102+0000 7f110d400700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f110810d310 con 0x7f1108103090 2024-04-28T12:41:16.099 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.102+0000 7f1106800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1108103090 0x7f11081a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:16.099 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.103+0000 7f1106800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1108103090 0x7f11081a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:38910/0 (socket says 172.21.15.84:38910) 2024-04-28T12:41:16.099 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.103+0000 7f1106800700 1 -- 172.21.15.84:0/3560675862 learned_addr learned my addr 172.21.15.84:0/3560675862 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:41:16.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.103+0000 7f1106800700 1 -- 172.21.15.84:0/3560675862 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11081a0c60 con 0x7f1108103090 2024-04-28T12:41:16.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.103+0000 7f1106800700 1 --2- 172.21.15.84:0/3560675862 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1108103090 0x7f11081a0720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f10f8006010 tx=0x7f10f8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:16.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.103+0000 7f10ff400700 1 -- 172.21.15.84:0/3560675862 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10f8026070 con 0x7f1108103090 2024-04-28T12:41:16.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.103+0000 7f10ff400700 1 -- 172.21.15.84:0/3560675862 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f800bba0 con 0x7f1108103090 2024-04-28T12:41:16.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.103+0000 7f110d400700 1 -- 172.21.15.84:0/3560675862 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11081a0f50 con 0x7f1108103090 2024-04-28T12:41:16.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.104+0000 7f10ff400700 1 -- 172.21.15.84:0/3560675862 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10f8003910 con 0x7f1108103090 2024-04-28T12:41:16.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.104+0000 7f110d400700 1 -- 172.21.15.84:0/3560675862 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f110819a8e0 con 0x7f1108103090 2024-04-28T12:41:16.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.105+0000 7f110d400700 1 -- 172.21.15.84:0/3560675862 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1108104570 con 0x7f1108103090 2024-04-28T12:41:16.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.105+0000 7f10ff400700 1 -- 172.21.15.84:0/3560675862 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f10f8029070 con 0x7f1108103090 2024-04-28T12:41:16.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.105+0000 7f10ff400700 1 --2- 172.21.15.84:0/3560675862 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f10ec0396b0 0x7f10ec03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:16.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.105+0000 7f10ff400700 1 -- 172.21.15.84:0/3560675862 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f10f801e070 con 0x7f1108103090 2024-04-28T12:41:16.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.106+0000 7f1105e00700 1 --2- 172.21.15.84:0/3560675862 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f10ec0396b0 0x7f10ec03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:16.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.106+0000 7f1105e00700 1 --2- 172.21.15.84:0/3560675862 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f10ec0396b0 0x7f10ec03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f10f4006fd0 tx=0x7f10f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:16.108 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.111+0000 7f10ff400700 1 -- 172.21.15.84:0/3560675862 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f10f8013070 con 0x7f1108103090 2024-04-28T12:41:16.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:16 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:16.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:16 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:16.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:16 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:16.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:16 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:16.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:16 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi179", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:41:16.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:16 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:16.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:16 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:16.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:16 smithi084 ceph-mon[28081]: Updating smithi179:/etc/ceph/ceph.conf 2024-04-28T12:41:16.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:16 smithi084 ceph-mon[28081]: Updating smithi179:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:16.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:16 smithi084 ceph-mon[28081]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:16.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:16.352+0000 7f110d400700 1 -- 172.21.15.84:0/3560675862 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f110819be10 con 0x7f1108103090 2024-04-28T12:41:17.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:17.297+0000 7f10ff400700 1 -- 172.21.15.84:0/3560675862 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f10f802ade0 con 0x7f1108103090 2024-04-28T12:41:17.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:17.302+0000 7f110d400700 1 -- 172.21.15.84:0/3560675862 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f10ec0396b0 msgr2=0x7f10ec03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:17.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:17.302+0000 7f110d400700 1 --2- 172.21.15.84:0/3560675862 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f10ec0396b0 0x7f10ec03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f10f4006fd0 tx=0x7f10f4006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:17.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:17.302+0000 7f110d400700 1 -- 172.21.15.84:0/3560675862 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1108103090 msgr2=0x7f11081a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:17.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:17.302+0000 7f110d400700 1 --2- 172.21.15.84:0/3560675862 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1108103090 0x7f11081a0720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f10f8006010 tx=0x7f10f8005780 comp rx=0 tx=0).stop 2024-04-28T12:41:17.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:17.302+0000 7f110d400700 1 -- 172.21.15.84:0/3560675862 shutdown_connections 2024-04-28T12:41:17.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:17.302+0000 7f110d400700 1 --2- 172.21.15.84:0/3560675862 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f10ec0396b0 0x7f10ec03bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:17.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:17.302+0000 7f110d400700 1 --2- 172.21.15.84:0/3560675862 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1108103090 0x7f11081a0720 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:17.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:17.302+0000 7f110d400700 1 -- 172.21.15.84:0/3560675862 >> 172.21.15.84:0/3560675862 conn(0x7f11080fe960 msgr2=0x7f1108075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:17.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:17.303+0000 7f110d400700 1 -- 172.21.15.84:0/3560675862 shutdown_connections 2024-04-28T12:41:17.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:17.303+0000 7f110d400700 1 -- 172.21.15.84:0/3560675862 wait complete. 2024-04-28T12:41:17.301 INFO:teuthology.orchestra.run.smithi084.stderr:adjusted tunables profile to default 2024-04-28T12:41:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:17 smithi084 ceph-mon[28081]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:17 smithi084 ceph-mon[28081]: Updating smithi179:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T12:41:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:17 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3560675862' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-28T12:41:18.052 INFO:tasks.cephadm:Adding mon.a on smithi084 2024-04-28T12:41:18.052 INFO:tasks.cephadm:Adding mon.b on smithi161 2024-04-28T12:41:18.052 INFO:tasks.cephadm:Adding mon.c on smithi179 2024-04-28T12:41:18.053 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi084:172.21.15.84=a;smithi161:172.21.15.161=b;smithi179:172.21.15.179=c' 2024-04-28T12:41:18.289 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:18.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:18 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3560675862' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-28T12:41:18.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:18 smithi084 ceph-mon[28081]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:41:18.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:18 smithi084 ceph-mon[28081]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:19.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.902+0000 7efd0761f700 1 -- 172.21.15.179:0/690374593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efd00107e50 msgr2=0x7efd00108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:19.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.902+0000 7efd0761f700 1 --2- 172.21.15.179:0/690374593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efd00107e50 0x7efd00108230 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009910 tx=0x7efcf0009c20 comp rx=0 tx=0).stop 2024-04-28T12:41:19.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.902+0000 7efd0761f700 1 -- 172.21.15.179:0/690374593 shutdown_connections 2024-04-28T12:41:19.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.902+0000 7efd0761f700 1 --2- 172.21.15.179:0/690374593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efd00107e50 0x7efd00108230 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:19.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.902+0000 7efd0761f700 1 -- 172.21.15.179:0/690374593 >> 172.21.15.179:0/690374593 conn(0x7efd000fcb90 msgr2=0x7efd000fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:19.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.902+0000 7efd0761f700 1 -- 172.21.15.179:0/690374593 shutdown_connections 2024-04-28T12:41:19.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.902+0000 7efd0761f700 1 -- 172.21.15.179:0/690374593 wait complete. 2024-04-28T12:41:19.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.902+0000 7efd0761f700 1 Processor -- start 2024-04-28T12:41:19.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.902+0000 7efd0761f700 1 -- start start 2024-04-28T12:41:19.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.903+0000 7efd0761f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efd00107e50 0x7efd001028a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:19.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.903+0000 7efd0761f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7efd0010d350 con 0x7efd00107e50 2024-04-28T12:41:19.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.903+0000 7efd053bb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efd00107e50 0x7efd001028a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:19.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.903+0000 7efd053bb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efd00107e50 0x7efd001028a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.179:59548/0 (socket says 172.21.15.179:59548) 2024-04-28T12:41:19.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.903+0000 7efd053bb700 1 -- 172.21.15.179:0/2101488912 learned_addr learned my addr 172.21.15.179:0/2101488912 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-04-28T12:41:19.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.904+0000 7efd053bb700 1 -- 172.21.15.179:0/2101488912 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd00104790 con 0x7efd00107e50 2024-04-28T12:41:19.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.904+0000 7efd053bb700 1 --2- 172.21.15.179:0/2101488912 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efd00107e50 0x7efd001028a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efcf00056d0 tx=0x7efcf00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:19.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.905+0000 7efcf67fc700 1 -- 172.21.15.179:0/2101488912 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcf0026070 con 0x7efd00107e50 2024-04-28T12:41:19.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.905+0000 7efd0761f700 1 -- 172.21.15.179:0/2101488912 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd00102de0 con 0x7efd00107e50 2024-04-28T12:41:19.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.905+0000 7efcf67fc700 1 -- 172.21.15.179:0/2101488912 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efcf000bc00 con 0x7efd00107e50 2024-04-28T12:41:19.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.905+0000 7efd0761f700 1 -- 172.21.15.179:0/2101488912 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd001032a0 con 0x7efd00107e50 2024-04-28T12:41:19.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.905+0000 7efcf67fc700 1 -- 172.21.15.179:0/2101488912 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcf0030060 con 0x7efd00107e50 2024-04-28T12:41:19.903 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.906+0000 7efd0761f700 1 -- 172.21.15.179:0/2101488912 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd00102350 con 0x7efd00107e50 2024-04-28T12:41:19.903 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.906+0000 7efcf67fc700 1 -- 172.21.15.179:0/2101488912 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7efcf0029070 con 0x7efd00107e50 2024-04-28T12:41:19.903 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.907+0000 7efcf67fc700 1 --2- 172.21.15.179:0/2101488912 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7efcec039650 0x7efcec03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:19.904 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.907+0000 7efcf67fc700 1 -- 172.21.15.179:0/2101488912 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efcf001e070 con 0x7efd00107e50 2024-04-28T12:41:19.904 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.907+0000 7efd04bba700 1 --2- 172.21.15.179:0/2101488912 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7efcec039650 0x7efcec03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:19.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.908+0000 7efd04bba700 1 --2- 172.21.15.179:0/2101488912 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7efcec039650 0x7efcec03bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efcfc006fd0 tx=0x7efcfc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:19.920 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:19.923+0000 7efcf67fc700 1 -- 172.21.15.179:0/2101488912 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efcf0013070 con 0x7efd00107e50 2024-04-28T12:41:20.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:20.130+0000 7efd0761f700 1 -- 172.21.15.179:0/2101488912 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi084:172.21.15.84=a;smithi161:172.21.15.161=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}) -- 0x7efd00103e40 con 0x7efcec039650 2024-04-28T12:41:20.142 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:20.145+0000 7efcf67fc700 1 -- 172.21.15.179:0/2101488912 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7efd00103e40 con 0x7efcec039650 2024-04-28T12:41:20.142 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mon update... 2024-04-28T12:41:20.144 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:20.147+0000 7efd0761f700 1 -- 172.21.15.179:0/2101488912 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7efcec039650 msgr2=0x7efcec03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:20.144 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:20.147+0000 7efd0761f700 1 --2- 172.21.15.179:0/2101488912 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7efcec039650 0x7efcec03bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efcfc006fd0 tx=0x7efcfc006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:20.144 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:20.147+0000 7efd0761f700 1 -- 172.21.15.179:0/2101488912 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efd00107e50 msgr2=0x7efd001028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:20.144 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:20.147+0000 7efd0761f700 1 --2- 172.21.15.179:0/2101488912 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efd00107e50 0x7efd001028a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efcf00056d0 tx=0x7efcf00057b0 comp rx=0 tx=0).stop 2024-04-28T12:41:20.144 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:20.147+0000 7efd0761f700 1 -- 172.21.15.179:0/2101488912 shutdown_connections 2024-04-28T12:41:20.144 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:20.147+0000 7efd0761f700 1 --2- 172.21.15.179:0/2101488912 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7efcec039650 0x7efcec03bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:20.144 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:20.147+0000 7efd0761f700 1 --2- 172.21.15.179:0/2101488912 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efd00107e50 0x7efd001028a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:20.145 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:20.147+0000 7efd0761f700 1 -- 172.21.15.179:0/2101488912 >> 172.21.15.179:0/2101488912 conn(0x7efd000fcb90 msgr2=0x7efd0010aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:20.145 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:20.147+0000 7efd0761f700 1 -- 172.21.15.179:0/2101488912 shutdown_connections 2024-04-28T12:41:20.145 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:20.147+0000 7efd0761f700 1 -- 172.21.15.179:0/2101488912 wait complete. 2024-04-28T12:41:20.746 DEBUG:teuthology.orchestra.run.smithi161:mon.b> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.b.service 2024-04-28T12:41:20.752 DEBUG:teuthology.orchestra.run.smithi179:mon.c> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.c.service 2024-04-28T12:41:20.754 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-28T12:41:20.754 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-28T12:41:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:20 smithi084 ceph-mon[28081]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:20 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:20 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:20 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:20 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:20.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:20 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:20.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:20 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:20.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:20 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:21.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:21 smithi084 ceph-mon[28081]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi084:172.21.15.84=a;smithi161:172.21.15.161=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:21.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:21 smithi084 ceph-mon[28081]: Saving service mon spec with placement smithi084:172.21.15.84=a;smithi161:172.21.15.161=b;smithi179:172.21.15.179=c;count:3 2024-04-28T12:41:21.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:21 smithi084 ceph-mon[28081]: Deploying daemon mon.c on smithi179 2024-04-28T12:41:22.495 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:22.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:22 smithi084 ceph-mon[28081]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:24.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:23 smithi179 systemd[1]: Starting Ceph mon.c for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:41:24.598 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.602+0000 7f6e41f2b700 1 -- 172.21.15.179:0/1993686101 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e3c103090 msgr2=0x7f6e3c103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:24.599 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.602+0000 7f6e41f2b700 1 --2- 172.21.15.179:0/1993686101 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e3c103090 0x7f6e3c103470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400b0a0 tx=0x7f6e3400b3b0 comp rx=0 tx=0).stop 2024-04-28T12:41:24.599 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.602+0000 7f6e41f2b700 1 -- 172.21.15.179:0/1993686101 shutdown_connections 2024-04-28T12:41:24.599 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.602+0000 7f6e41f2b700 1 --2- 172.21.15.179:0/1993686101 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e3c103090 0x7f6e3c103470 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:24.599 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.602+0000 7f6e41f2b700 1 -- 172.21.15.179:0/1993686101 >> 172.21.15.179:0/1993686101 conn(0x7f6e3c0fe960 msgr2=0x7f6e3c100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:24.599 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.602+0000 7f6e41f2b700 1 -- 172.21.15.179:0/1993686101 shutdown_connections 2024-04-28T12:41:24.599 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.602+0000 7f6e41f2b700 1 -- 172.21.15.179:0/1993686101 wait complete. 2024-04-28T12:41:24.599 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.602+0000 7f6e41f2b700 1 Processor -- start 2024-04-28T12:41:24.599 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.602+0000 7f6e41f2b700 1 -- start start 2024-04-28T12:41:24.599 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.603+0000 7f6e41f2b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e3c103090 0x7f6e3c1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:24.599 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.603+0000 7f6e41f2b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c10d310 con 0x7f6e3c103090 2024-04-28T12:41:24.600 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.603+0000 7f6e3b7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e3c103090 0x7f6e3c1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:24.600 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.603+0000 7f6e3b7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e3c103090 0x7f6e3c1a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.179:50948/0 (socket says 172.21.15.179:50948) 2024-04-28T12:41:24.600 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.603+0000 7f6e3b7fe700 1 -- 172.21.15.179:0/2032076237 learned_addr learned my addr 172.21.15.179:0/2032076237 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-04-28T12:41:24.600 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.604+0000 7f6e3b7fe700 1 -- 172.21.15.179:0/2032076237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e3c1a0be0 con 0x7f6e3c103090 2024-04-28T12:41:24.601 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.604+0000 7f6e3b7fe700 1 --2- 172.21.15.179:0/2032076237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e3c103090 0x7f6e3c1a06a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400d010 tx=0x7f6e34003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:24.601 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.605+0000 7f6e38ff9700 1 -- 172.21.15.179:0/2032076237 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6e34021020 con 0x7f6e3c103090 2024-04-28T12:41:24.601 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.605+0000 7f6e41f2b700 1 -- 172.21.15.179:0/2032076237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e3c1a0e70 con 0x7f6e3c103090 2024-04-28T12:41:24.601 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.605+0000 7f6e41f2b700 1 -- 172.21.15.179:0/2032076237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e3c19a820 con 0x7f6e3c103090 2024-04-28T12:41:24.601 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.605+0000 7f6e38ff9700 1 -- 172.21.15.179:0/2032076237 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e340044b0 con 0x7f6e3c103090 2024-04-28T12:41:24.601 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.605+0000 7f6e38ff9700 1 -- 172.21.15.179:0/2032076237 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6e34017040 con 0x7f6e3c103090 2024-04-28T12:41:24.601 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.606+0000 7f6e41f2b700 1 -- 172.21.15.179:0/2032076237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e3c104570 con 0x7f6e3c103090 2024-04-28T12:41:24.602 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.606+0000 7f6e38ff9700 1 -- 172.21.15.179:0/2032076237 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6e34024020 con 0x7f6e3c103090 2024-04-28T12:41:24.602 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.606+0000 7f6e38ff9700 1 --2- 172.21.15.179:0/2032076237 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6e240396a0 0x7f6e2403bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:24.602 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.606+0000 7f6e38ff9700 1 -- 172.21.15.179:0/2032076237 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6e3400f050 con 0x7f6e3c103090 2024-04-28T12:41:24.603 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.607+0000 7f6e3affd700 1 --2- 172.21.15.179:0/2032076237 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6e240396a0 0x7f6e2403bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:24.603 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.608+0000 7f6e3affd700 1 --2- 172.21.15.179:0/2032076237 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6e240396a0 0x7f6e2403bb60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6e30006fd0 tx=0x7f6e30006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:24.618 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.622+0000 7f6e38ff9700 1 -- 172.21.15.179:0/2032076237 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6e3402b030 con 0x7f6e3c103090 2024-04-28T12:41:24.904 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.908+0000 7f6e41f2b700 1 -- 172.21.15.179:0/2032076237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6e3c064a80 con 0x7f6e3c103090 2024-04-28T12:41:24.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.909+0000 7f6e38ff9700 1 -- 172.21.15.179:0/2032076237 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f6e34025bf0 con 0x7f6e3c103090 2024-04-28T12:41:24.905 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:41:24.905 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":1,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","modified":"2024-04-28T12:39:18.014667Z","created":"2024-04-28T12:39:18.014667Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-28T12:41:24.907 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.911+0000 7f6e41f2b700 1 -- 172.21.15.179:0/2032076237 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6e240396a0 msgr2=0x7f6e2403bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:24.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.911+0000 7f6e41f2b700 1 --2- 172.21.15.179:0/2032076237 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6e240396a0 0x7f6e2403bb60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6e30006fd0 tx=0x7f6e30006e40 comp rx=0 tx=0).stop 2024-04-28T12:41:24.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.911+0000 7f6e41f2b700 1 -- 172.21.15.179:0/2032076237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e3c103090 msgr2=0x7f6e3c1a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:24.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.911+0000 7f6e41f2b700 1 --2- 172.21.15.179:0/2032076237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e3c103090 0x7f6e3c1a06a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400d010 tx=0x7f6e34003ce0 comp rx=0 tx=0).stop 2024-04-28T12:41:24.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.911+0000 7f6e41f2b700 1 -- 172.21.15.179:0/2032076237 shutdown_connections 2024-04-28T12:41:24.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.911+0000 7f6e41f2b700 1 --2- 172.21.15.179:0/2032076237 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6e240396a0 0x7f6e2403bb60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:24.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.911+0000 7f6e41f2b700 1 --2- 172.21.15.179:0/2032076237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e3c103090 0x7f6e3c1a06a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:24.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.911+0000 7f6e41f2b700 1 -- 172.21.15.179:0/2032076237 >> 172.21.15.179:0/2032076237 conn(0x7f6e3c0fe960 msgr2=0x7f6e3c075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:24.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.911+0000 7f6e41f2b700 1 -- 172.21.15.179:0/2032076237 shutdown_connections 2024-04-28T12:41:24.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:24.911+0000 7f6e41f2b700 1 -- 172.21.15.179:0/2032076237 wait complete. 2024-04-28T12:41:24.909 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 1 2024-04-28T12:41:24.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:24 smithi084 ceph-mon[28081]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:24.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 podman[62975]: 2024-04-28 12:41:24.605158361 +0000 UTC m=+0.020258161 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:41:24.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 podman[62975]: 2024-04-28 12:41:24.715693323 +0000 UTC m=+0.130793107 container create 39e2a04435a15370a1c8e93d533112bc7df7d3aee0100f20b9b58aead652f7d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-28T12:41:24.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 podman[62975]: 2024-04-28 12:41:24.94294812 +0000 UTC m=+0.358047912 container init 39e2a04435a15370a1c8e93d533112bc7df7d3aee0100f20b9b58aead652f7d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-c, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-28T12:41:24.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 podman[62975]: 2024-04-28 12:41:24.946410629 +0000 UTC m=+0.361510414 container start 39e2a04435a15370a1c8e93d533112bc7df7d3aee0100f20b9b58aead652f7d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0) 2024-04-28T12:41:24.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: set uid:gid to 167:167 (ceph:ceph) 2024-04-28T12:41:24.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-28T12:41:24.966 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: pidfile_write: ignore empty --pid-file 2024-04-28T12:41:25.326 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: load: jerasure load: lrc 2024-04-28T12:41:25.326 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: RocksDB version: 7.9.2 2024-04-28T12:41:25.326 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Git sha 0 2024-04-28T12:41:25.326 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-28T12:41:25.326 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: DB SUMMARY 2024-04-28T12:41:25.326 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: DB Session ID: NXO17CBC79OFLIQHN4R2 2024-04-28T12:41:25.326 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: CURRENT file: CURRENT 2024-04-28T12:41:25.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: IDENTITY file: IDENTITY 2024-04-28T12:41:25.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-28T12:41:25.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-28T12:41:25.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-28T12:41:25.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.error_if_exists: 0 2024-04-28T12:41:25.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.create_if_missing: 0 2024-04-28T12:41:25.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.paranoid_checks: 1 2024-04-28T12:41:25.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-28T12:41:25.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-28T12:41:25.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-28T12:41:25.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.env: 0x55e84068acc0 2024-04-28T12:41:25.327 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.fs: PosixFileSystem 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.info_log: 0x55e84141bd80 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_file_opening_threads: 16 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.statistics: (nil) 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.use_fsync: 0 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_log_file_size: 0 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.keep_log_file_num: 1000 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.recycle_log_file_num: 0 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.allow_fallocate: 1 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.allow_mmap_reads: 0 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.allow_mmap_writes: 0 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.use_direct_reads: 0 2024-04-28T12:41:25.328 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.create_missing_column_families: 0 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.db_log_dir: 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.wal_dir: 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.advise_random_on_open: 1 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.db_write_buffer_size: 0 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.write_buffer_manager: 0x55e8413ed0e0 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-28T12:41:25.329 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.rate_limiter: (nil) 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.wal_recovery_mode: 2 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.enable_thread_tracking: 0 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.enable_pipelined_write: 0 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.unordered_write: 0 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.row_cache: None 2024-04-28T12:41:25.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.wal_filter: None 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.allow_ingest_behind: 0 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.two_write_queues: 0 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.manual_wal_flush: 0 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.wal_compression: 0 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.atomic_flush: 0 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.log_readahead_size: 0 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.best_efforts_recovery: 0 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-28T12:41:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-28T12:41:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.allow_data_in_errors: 0 2024-04-28T12:41:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.db_host_id: __hostname__ 2024-04-28T12:41:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-28T12:41:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_background_jobs: 2 2024-04-28T12:41:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_background_compactions: -1 2024-04-28T12:41:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_subcompactions: 1 2024-04-28T12:41:25.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-28T12:41:25.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-28T12:41:25.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-28T12:41:25.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_total_wal_size: 0 2024-04-28T12:41:25.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-28T12:41:25.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-28T12:41:25.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-28T12:41:25.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-28T12:41:25.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_open_files: -1 2024-04-28T12:41:25.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bytes_per_sync: 0 2024-04-28T12:41:25.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-28T12:41:25.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-28T12:41:25.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_readahead_size: 0 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_background_flushes: -1 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Compression algorithms supported: 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: kZSTD supported: 0 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: kXpressCompression supported: 0 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: kLZ4HCCompression supported: 1 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: kZlibCompression supported: 1 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: kSnappyCompression supported: 1 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: kLZ4Compression supported: 1 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: kBZip2Compression supported: 0 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-28T12:41:25.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.merge_operator: 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_filter: None 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_filter_factory: None 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.sst_partitioner_factory: None 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e84141bf40) 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout: cache_index_and_filter_blocks: 1 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout: pin_top_level_index_and_filter: 1 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout: index_type: 0 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout: data_block_index_type: 0 2024-04-28T12:41:25.336 INFO:journalctl@ceph.mon.c.smithi179.stdout: index_shortening: 1 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: checksum: 4 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: no_block_cache: 0 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache: 0x55e84141f1f0 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache_name: BinnedLRUCache 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache_options: 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: capacity : 536870912 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: num_shard_bits : 4 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: strict_capacity_limit : 0 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: high_pri_pool_ratio: 0.000 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_cache_compressed: (nil) 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: persistent_cache: (nil) 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_size: 4096 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_size_deviation: 10 2024-04-28T12:41:25.337 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_restart_interval: 16 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: index_block_restart_interval: 1 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: metadata_block_size: 4096 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: partition_filters: 0 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: use_delta_encoding: 1 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: filter_policy: bloomfilter 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: whole_key_filtering: 1 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: verify_compression: 0 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: read_amp_bytes_per_bit: 0 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: format_version: 5 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: enable_index_compression: 1 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: block_align: 0 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: max_auto_readahead_size: 262144 2024-04-28T12:41:25.338 INFO:journalctl@ceph.mon.c.smithi179.stdout: prepopulate_block_cache: 0 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout: initial_auto_readahead_size: 8192 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout: num_file_reads_for_auto_readahead: 2 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.write_buffer_size: 33554432 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_write_buffer_number: 2 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compression: NoCompression 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bottommost_compression: Disabled 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.prefix_extractor: nullptr 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.num_levels: 7 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-28T12:41:25.339 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compression_opts.level: 32767 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compression_opts.strategy: 0 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compression_opts.enabled: false 2024-04-28T12:41:25.340 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.target_file_size_base: 67108864 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-28T12:41:25.341 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.arena_block_size: 1048576 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.disable_auto_compactions: 0 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-28T12:41:25.342 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.inplace_update_support: 0 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.bloom_locality: 0 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.max_successive_merges: 0 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-28T12:41:25.343 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.paranoid_file_checks: 0 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.force_consistency_checks: 1 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.report_bg_io_stats: 0 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.ttl: 2592000 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.enable_blob_files: false 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.min_blob_size: 0 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.blob_file_size: 268435456 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-28T12:41:25.344 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.blob_file_starting_level: 0 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 82b14bc6-07bf-450b-9578-efa7db27a70f 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714308084976879, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714308084977286, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714308084, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "82b14bc6-07bf-450b-9578-efa7db27a70f", "db_session_id": "NXO17CBC79OFLIQHN4R2", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714308084977366, "job": 1, "event": "recovery_finished"} 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e84153c000 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: DB pointer 0x55e841528000 2024-04-28T12:41:25.345 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** DB Stats ** 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** Compaction Stats [default] ** 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T12:41:25.346 INFO:journalctl@ceph.mon.c.smithi179.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** Compaction Stats [default] ** 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T12:41:25.347 INFO:journalctl@ceph.mon.c.smithi179.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout: Block cache BinnedLRUCache@0x55e84141f1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: using public_addr v2:172.21.15.179:0/0 -> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] 2024-04-28T12:41:25.348 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: starting mon.c rank -1 at public addrs [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] at bind addrs [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:24 smithi179 ceph-mon[63029]: mon.c@-1(???) e0 preinit fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).mds e1 new map 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout: e1 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout: legacy client fscid: -1 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout: 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout: No filesystems configured 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-28T12:41:25.349 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Added host smithi161 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.350 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Updating smithi161:/etc/ceph/ceph.conf 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Updating smithi161:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Updating smithi161:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T12:41:25.351 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Deploying cephadm binary to smithi179 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Added host smithi179 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.352 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi179", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Updating smithi179:/etc/ceph/ceph.conf 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Updating smithi179:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Updating smithi179:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.353 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3560675862' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3560675862' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi084:172.21.15.84=a;smithi161:172.21.15.161=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Saving service mon spec with placement smithi084:172.21.15.84=a;smithi161:172.21.15.161=b;smithi179:172.21.15.179=c;count:3 2024-04-28T12:41:25.354 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: Deploying daemon mon.c on smithi179 2024-04-28T12:41:25.355 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.355 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:25.355 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 ceph-mon[63029]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-28T12:41:25.355 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 bash[62975]: 39e2a04435a15370a1c8e93d533112bc7df7d3aee0100f20b9b58aead652f7d7 2024-04-28T12:41:25.355 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:25 smithi179 systemd[1]: Started Ceph mon.c for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:41:26.595 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-28T12:41:26.595 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-28T12:41:26.839 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:41:30.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:41:30.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: mon.a calling monitor election 2024-04-28T12:41:30.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:30.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:30.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:30.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:30.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: mon.c calling monitor election 2024-04-28T12:41:30.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:30.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:30.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: monmap epoch 2 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: last_changed 2024-04-28T12:41:25.016292+0000 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: created 2024-04-28T12:39:18.014667+0000 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: min_mon_release 19 (squid) 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: election_strategy: 1 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: fsmap 2024-04-28T12:41:30.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:41:30.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: mgrmap e13: x(active, since 60s) 2024-04-28T12:41:30.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: overall HEALTH_OK 2024-04-28T12:41:30.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:30.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:30.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:30.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:30.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: mon.a calling monitor election 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: mon.c calling monitor election 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: monmap epoch 2 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:30.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: last_changed 2024-04-28T12:41:25.016292+0000 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: created 2024-04-28T12:39:18.014667+0000 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: min_mon_release 19 (squid) 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: election_strategy: 1 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: fsmap 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: mgrmap e13: x(active, since 60s) 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: overall HEALTH_OK 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:30.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:31.244 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.247+0000 7f4b78bc9700 1 -- 172.21.15.179:0/2461599377 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b5c0035c0 msgr2=0x7f4b5c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:31.244 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.247+0000 7f4b78bc9700 1 --2- 172.21.15.179:0/2461599377 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b5c0035c0 0x7f4b5c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b74102e70 tx=0x7f4b6c009e80 comp rx=0 tx=0).stop 2024-04-28T12:41:31.244 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.247+0000 7f4b78bc9700 1 -- 172.21.15.179:0/2461599377 shutdown_connections 2024-04-28T12:41:31.244 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.247+0000 7f4b78bc9700 1 --2- 172.21.15.179:0/2461599377 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b5c0035c0 0x7f4b5c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:31.244 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.247+0000 7f4b78bc9700 1 -- 172.21.15.179:0/2461599377 >> 172.21.15.179:0/2461599377 conn(0x7f4b740fea30 msgr2=0x7f4b74100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:31.244 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.247+0000 7f4b78bc9700 1 -- 172.21.15.179:0/2461599377 shutdown_connections 2024-04-28T12:41:31.244 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.247+0000 7f4b78bc9700 1 -- 172.21.15.179:0/2461599377 wait complete. 2024-04-28T12:41:31.245 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.248+0000 7f4b78bc9700 1 Processor -- start 2024-04-28T12:41:31.245 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b78bc9700 1 -- start start 2024-04-28T12:41:31.246 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b78bc9700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b5c0035c0 0x7f4b7419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:31.246 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b78bc9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b7419ebe0 0x7f4b74198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:31.246 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b78bc9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7410ae90 con 0x7f4b7419ebe0 2024-04-28T12:41:31.246 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b78bc9700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7410ad10 con 0x7f4b5c0035c0 2024-04-28T12:41:31.246 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b7259c700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b5c0035c0 0x7f4b7419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:31.246 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b7259c700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b5c0035c0 0x7f4b7419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:44536/0 (socket says 172.21.15.179:44536) 2024-04-28T12:41:31.246 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b7259c700 1 -- 172.21.15.179:0/3854096541 learned_addr learned my addr 172.21.15.179:0/3854096541 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-04-28T12:41:31.246 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b7259c700 1 -- 172.21.15.179:0/3854096541 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b5c0035c0 msgr2=0x7f4b7419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-28T12:41:31.246 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b7259c700 1 -- 172.21.15.179:0/3854096541 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b5c0035c0 msgr2=0x7f4b7419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-28T12:41:31.246 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b7259c700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b5c0035c0 0x7f4b7419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-28T12:41:31.246 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b7259c700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b5c0035c0 0x7f4b7419e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:41:31.246 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.249+0000 7f4b71d9b700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b7419ebe0 0x7f4b74198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:31.247 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.250+0000 7f4b71d9b700 1 -- 172.21.15.179:0/3854096541 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b5c0035c0 msgr2=0x7f4b7419e6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:41:31.247 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.250+0000 7f4b71d9b700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b5c0035c0 0x7f4b7419e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:31.247 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.250+0000 7f4b71d9b700 1 -- 172.21.15.179:0/3854096541 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7419f220 con 0x7f4b7419ebe0 2024-04-28T12:41:31.247 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.250+0000 7f4b71d9b700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b7419ebe0 0x7f4b74198860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800b4d0 tx=0x7f4b6800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:31.248 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.251+0000 7f4b637fe700 1 -- 172.21.15.179:0/3854096541 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f4b68011630 con 0x7f4b7419ebe0 2024-04-28T12:41:31.248 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.251+0000 7f4b78bc9700 1 -- 172.21.15.179:0/3854096541 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b74198da0 con 0x7f4b7419ebe0 2024-04-28T12:41:31.248 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.251+0000 7f4b637fe700 1 -- 172.21.15.179:0/3854096541 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b68011c80 con 0x7f4b7419ebe0 2024-04-28T12:41:31.248 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.251+0000 7f4b637fe700 1 -- 172.21.15.179:0/3854096541 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f4b68010da0 con 0x7f4b7419ebe0 2024-04-28T12:41:31.248 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.251+0000 7f4b78bc9700 1 -- 172.21.15.179:0/3854096541 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b74199330 con 0x7f4b7419ebe0 2024-04-28T12:41:31.248 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.252+0000 7f4b78bc9700 1 -- 172.21.15.179:0/3854096541 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b74104680 con 0x7f4b7419ebe0 2024-04-28T12:41:31.249 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.252+0000 7f4b637fe700 1 -- 172.21.15.179:0/3854096541 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4b68018020 con 0x7f4b7419ebe0 2024-04-28T12:41:31.249 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.253+0000 7f4b637fe700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4b54039410 0x7f4b5403b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:31.249 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.253+0000 7f4b637fe700 1 -- 172.21.15.179:0/3854096541 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4b6804c200 con 0x7f4b7419ebe0 2024-04-28T12:41:31.249 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.253+0000 7f4b7259c700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4b54039410 0x7f4b5403b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:31.250 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.254+0000 7f4b7259c700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4b54039410 0x7f4b5403b8d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c007590 tx=0x7f4b6c0074e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:31.263 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.267+0000 7f4b637fe700 1 -- 172.21.15.179:0/3854096541 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4b6801f020 con 0x7f4b7419ebe0 2024-04-28T12:41:31.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:31 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:31.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:31 smithi179 ceph-mon[63029]: Deploying daemon mon.b on smithi161 2024-04-28T12:41:31.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:31 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:31.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:31 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:31.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:31 smithi084 ceph-mon[28081]: Deploying daemon mon.b on smithi161 2024-04-28T12:41:31.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:31 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:31.548 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.552+0000 7f4b78bc9700 1 -- 172.21.15.179:0/3854096541 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f4b74199d90 con 0x7f4b7419ebe0 2024-04-28T12:41:31.549 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.553+0000 7f4b637fe700 1 -- 172.21.15.179:0/3854096541 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f4b68010410 con 0x7f4b7419ebe0 2024-04-28T12:41:31.549 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:41:31.550 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":2,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","modified":"2024-04-28T12:41:25.016292Z","created":"2024-04-28T12:39:18.014667Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3300","nonce":0},{"type":"v1","addr":"172.21.15.179:6789","nonce":0}]},"addr":"172.21.15.179:6789/0","public_addr":"172.21.15.179:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-28T12:41:31.551 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.555+0000 7f4b78bc9700 1 -- 172.21.15.179:0/3854096541 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4b54039410 msgr2=0x7f4b5403b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:31.552 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.555+0000 7f4b78bc9700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4b54039410 0x7f4b5403b8d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c007590 tx=0x7f4b6c0074e0 comp rx=0 tx=0).stop 2024-04-28T12:41:31.552 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.555+0000 7f4b78bc9700 1 -- 172.21.15.179:0/3854096541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b7419ebe0 msgr2=0x7f4b74198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:31.552 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.555+0000 7f4b78bc9700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b7419ebe0 0x7f4b74198860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800b4d0 tx=0x7f4b6800b890 comp rx=0 tx=0).stop 2024-04-28T12:41:31.552 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.555+0000 7f4b78bc9700 1 -- 172.21.15.179:0/3854096541 shutdown_connections 2024-04-28T12:41:31.552 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.555+0000 7f4b78bc9700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4b54039410 0x7f4b5403b8d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:31.552 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.555+0000 7f4b78bc9700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b5c0035c0 0x7f4b7419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:31.552 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.555+0000 7f4b78bc9700 1 --2- 172.21.15.179:0/3854096541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b7419ebe0 0x7f4b74198860 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:31.552 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.555+0000 7f4b78bc9700 1 -- 172.21.15.179:0/3854096541 >> 172.21.15.179:0/3854096541 conn(0x7f4b740fea30 msgr2=0x7f4b740ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:31.552 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.555+0000 7f4b78bc9700 1 -- 172.21.15.179:0/3854096541 shutdown_connections 2024-04-28T12:41:31.552 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:31.556+0000 7f4b78bc9700 1 -- 172.21.15.179:0/3854096541 wait complete. 2024-04-28T12:41:31.553 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 2 2024-04-28T12:41:32.153 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:32 smithi179 ceph-mon[63029]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:32.154 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:32 smithi179 ceph-mon[63029]: from='client.? 172.21.15.179:0/3854096541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:41:32.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:32 smithi084 ceph-mon[28081]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:32.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:32 smithi084 ceph-mon[28081]: from='client.? 172.21.15.179:0/3854096541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:41:32.417 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:41:32 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:41:32.017+0000 7f01e2000700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-28T12:41:33.181 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-28T12:41:33.181 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-28T12:41:33.423 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:41:33.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:33 smithi161 podman[62369]: 2024-04-28 12:41:33.818386642 +0000 UTC m=+0.018057040 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:41:34.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 podman[62369]: 2024-04-28 12:41:34.130110003 +0000 UTC m=+0.329780391 container init 0411b65b5f435951d27b00c6bf7666905d59b29e34d9bfc83285fc8b6c093e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-28T12:41:34.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 podman[62369]: 2024-04-28 12:41:34.13325679 +0000 UTC m=+0.332927167 container start 0411b65b5f435951d27b00c6bf7666905d59b29e34d9bfc83285fc8b6c093e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=) 2024-04-28T12:41:34.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: set uid:gid to 167:167 (ceph:ceph) 2024-04-28T12:41:34.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-28T12:41:34.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pidfile_write: ignore empty --pid-file 2024-04-28T12:41:34.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: load: jerasure load: lrc 2024-04-28T12:41:34.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: RocksDB version: 7.9.2 2024-04-28T12:41:34.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Git sha 0 2024-04-28T12:41:34.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-28T12:41:34.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: DB SUMMARY 2024-04-28T12:41:34.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: DB Session ID: O3FJ194WGWBFMG6BB7V3 2024-04-28T12:41:34.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: CURRENT file: CURRENT 2024-04-28T12:41:34.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: IDENTITY file: IDENTITY 2024-04-28T12:41:34.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-28T12:41:34.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-28T12:41:34.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-28T12:41:34.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.error_if_exists: 0 2024-04-28T12:41:34.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.create_if_missing: 0 2024-04-28T12:41:34.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.paranoid_checks: 1 2024-04-28T12:41:34.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-28T12:41:34.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-28T12:41:34.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-28T12:41:34.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.env: 0x55cd3588ecc0 2024-04-28T12:41:34.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.fs: PosixFileSystem 2024-04-28T12:41:34.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.info_log: 0x55cd38077d80 2024-04-28T12:41:34.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_file_opening_threads: 16 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.statistics: (nil) 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.use_fsync: 0 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_log_file_size: 0 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.keep_log_file_num: 1000 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.recycle_log_file_num: 0 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.allow_fallocate: 1 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.allow_mmap_reads: 0 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.allow_mmap_writes: 0 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.use_direct_reads: 0 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-28T12:41:34.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.create_missing_column_families: 0 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.db_log_dir: 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.wal_dir: 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.advise_random_on_open: 1 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.db_write_buffer_size: 0 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.write_buffer_manager: 0x55cd380490e0 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-28T12:41:34.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-28T12:41:34.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.rate_limiter: (nil) 2024-04-28T12:41:34.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-28T12:41:34.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.wal_recovery_mode: 2 2024-04-28T12:41:34.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.enable_thread_tracking: 0 2024-04-28T12:41:34.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.enable_pipelined_write: 0 2024-04-28T12:41:34.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.unordered_write: 0 2024-04-28T12:41:34.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-28T12:41:34.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-28T12:41:34.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-28T12:41:34.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-28T12:41:34.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.row_cache: None 2024-04-28T12:41:34.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.wal_filter: None 2024-04-28T12:41:34.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-28T12:41:34.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.allow_ingest_behind: 0 2024-04-28T12:41:34.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.two_write_queues: 0 2024-04-28T12:41:34.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.manual_wal_flush: 0 2024-04-28T12:41:34.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.wal_compression: 0 2024-04-28T12:41:34.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.atomic_flush: 0 2024-04-28T12:41:34.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-28T12:41:34.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-28T12:41:34.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-28T12:41:34.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.log_readahead_size: 0 2024-04-28T12:41:34.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-28T12:41:34.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.best_efforts_recovery: 0 2024-04-28T12:41:34.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-28T12:41:34.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-28T12:41:34.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.allow_data_in_errors: 0 2024-04-28T12:41:34.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.db_host_id: __hostname__ 2024-04-28T12:41:34.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-28T12:41:34.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_background_jobs: 2 2024-04-28T12:41:34.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_background_compactions: -1 2024-04-28T12:41:34.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_subcompactions: 1 2024-04-28T12:41:34.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-28T12:41:34.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-28T12:41:34.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-28T12:41:34.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_total_wal_size: 0 2024-04-28T12:41:34.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-28T12:41:34.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-28T12:41:34.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-28T12:41:34.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-28T12:41:34.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_open_files: -1 2024-04-28T12:41:34.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bytes_per_sync: 0 2024-04-28T12:41:34.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_readahead_size: 0 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_background_flushes: -1 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Compression algorithms supported: 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: kZSTD supported: 0 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: kXpressCompression supported: 0 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: kLZ4HCCompression supported: 1 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: kZlibCompression supported: 1 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: kSnappyCompression supported: 1 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: kLZ4Compression supported: 1 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: kBZip2Compression supported: 0 2024-04-28T12:41:34.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.merge_operator: 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_filter: None 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_filter_factory: None 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.sst_partitioner_factory: None 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cd38077f40) 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout: cache_index_and_filter_blocks: 1 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-28T12:41:34.504 INFO:journalctl@ceph.mon.b.smithi161.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: pin_top_level_index_and_filter: 1 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: index_type: 0 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: data_block_index_type: 0 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: index_shortening: 1 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: checksum: 4 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: no_block_cache: 0 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_cache: 0x55cd3807b1f0 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_cache_name: BinnedLRUCache 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_cache_options: 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: capacity : 536870912 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: num_shard_bits : 4 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: strict_capacity_limit : 0 2024-04-28T12:41:34.505 INFO:journalctl@ceph.mon.b.smithi161.stdout: high_pri_pool_ratio: 0.000 2024-04-28T12:41:34.506 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_cache_compressed: (nil) 2024-04-28T12:41:34.506 INFO:journalctl@ceph.mon.b.smithi161.stdout: persistent_cache: (nil) 2024-04-28T12:41:34.506 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_size: 4096 2024-04-28T12:41:34.506 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_size_deviation: 10 2024-04-28T12:41:34.506 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_restart_interval: 16 2024-04-28T12:41:34.506 INFO:journalctl@ceph.mon.b.smithi161.stdout: index_block_restart_interval: 1 2024-04-28T12:41:34.506 INFO:journalctl@ceph.mon.b.smithi161.stdout: metadata_block_size: 4096 2024-04-28T12:41:34.506 INFO:journalctl@ceph.mon.b.smithi161.stdout: partition_filters: 0 2024-04-28T12:41:34.506 INFO:journalctl@ceph.mon.b.smithi161.stdout: use_delta_encoding: 1 2024-04-28T12:41:34.506 INFO:journalctl@ceph.mon.b.smithi161.stdout: filter_policy: bloomfilter 2024-04-28T12:41:34.506 INFO:journalctl@ceph.mon.b.smithi161.stdout: whole_key_filtering: 1 2024-04-28T12:41:34.506 INFO:journalctl@ceph.mon.b.smithi161.stdout: verify_compression: 0 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout: read_amp_bytes_per_bit: 0 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout: format_version: 5 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout: enable_index_compression: 1 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_align: 0 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout: max_auto_readahead_size: 262144 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout: prepopulate_block_cache: 0 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout: initial_auto_readahead_size: 8192 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout: num_file_reads_for_auto_readahead: 2 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.write_buffer_size: 33554432 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_write_buffer_number: 2 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compression: NoCompression 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bottommost_compression: Disabled 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.prefix_extractor: nullptr 2024-04-28T12:41:34.507 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.num_levels: 7 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-28T12:41:34.508 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compression_opts.level: 32767 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compression_opts.strategy: 0 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compression_opts.enabled: false 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-28T12:41:34.509 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.target_file_size_base: 67108864 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-28T12:41:34.510 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.arena_block_size: 1048576 2024-04-28T12:41:34.511 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-28T12:41:34.511 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-28T12:41:34.511 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.disable_auto_compactions: 0 2024-04-28T12:41:34.511 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-28T12:41:34.511 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-28T12:41:34.511 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-28T12:41:34.511 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-28T12:41:34.511 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-28T12:41:34.511 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-28T12:41:34.511 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-28T12:41:34.511 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-28T12:41:34.511 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.inplace_update_support: 0 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.bloom_locality: 0 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.max_successive_merges: 0 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.paranoid_file_checks: 0 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.force_consistency_checks: 1 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.report_bg_io_stats: 0 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.ttl: 2592000 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-28T12:41:34.512 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.enable_blob_files: false 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.min_blob_size: 0 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.blob_file_size: 268435456 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.blob_file_starting_level: 0 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7ee2b45f-fcbc-4209-8499-91c814ffeeb8 2024-04-28T12:41:34.513 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714308094163674, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714308094164238, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714308094, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7ee2b45f-fcbc-4209-8499-91c814ffeeb8", "db_session_id": "O3FJ194WGWBFMG6BB7V3", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714308094164342, "job": 1, "event": "recovery_finished"} 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cd38198000 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: DB pointer 0x55cd38184000 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout: ** DB Stats ** 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-28T12:41:34.514 INFO:journalctl@ceph.mon.b.smithi161.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: ** Compaction Stats [default] ** 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: ** Compaction Stats [default] ** 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-28T12:41:34.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: Block cache BinnedLRUCache@0x55cd3807b1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-28T12:41:34.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: using public_addr v2:172.21.15.161:0/0 -> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: starting mon.b rank -1 at public addrs [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] at bind addrs [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(???) e0 preinit fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).mds e1 new map 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout: e1 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout: legacy client fscid: -1 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout: No filesystems configured 2024-04-28T12:41:34.517 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Added host smithi161 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Updating smithi161:/etc/ceph/ceph.conf 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Updating smithi161:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Updating smithi161:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Deploying cephadm binary to smithi179 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Added host smithi179 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi179", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Updating smithi179:/etc/ceph/ceph.conf 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Updating smithi179:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:34.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Updating smithi179:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3560675862' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3560675862' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:34.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi084:172.21.15.84=a;smithi161:172.21.15.161=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Saving service mon spec with placement smithi084:172.21.15.84=a;smithi161:172.21.15.161=b;smithi179:172.21.15.179=c;count:3 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Deploying daemon mon.c on smithi179 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.a calling monitor election 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.c calling monitor election 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: monmap epoch 2 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: last_changed 2024-04-28T12:41:25.016292+0000 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: created 2024-04-28T12:39:18.014667+0000 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: min_mon_release 19 (squid) 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: election_strategy: 1 2024-04-28T12:41:34.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: fsmap 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mgrmap e13: x(active, since 60s) 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: overall HEALTH_OK 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: Deploying daemon mon.b on smithi161 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: from='client.? 172.21.15.179:0/3854096541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:41:34.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 ceph-mon[62387]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-28T12:41:34.526 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 bash[62369]: 0411b65b5f435951d27b00c6bf7666905d59b29e34d9bfc83285fc8b6c093e25 2024-04-28T12:41:34.526 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:34 smithi161 systemd[1]: Started Ceph mon.b for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:41:39.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.288+0000 7f7408237700 1 -- 172.21.15.179:0/3055509222 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f73e80035c0 msgr2=0x7f73e8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:39.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.288+0000 7f7408237700 1 --2- 172.21.15.179:0/3055509222 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f73e80035c0 0x7f73e8005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7400079550 tx=0x7f73f8009e80 comp rx=0 tx=0).stop 2024-04-28T12:41:39.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.288+0000 7f7408237700 1 -- 172.21.15.179:0/3055509222 shutdown_connections 2024-04-28T12:41:39.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.288+0000 7f7408237700 1 --2- 172.21.15.179:0/3055509222 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f73e80035c0 0x7f73e8005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:39.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.288+0000 7f7408237700 1 -- 172.21.15.179:0/3055509222 >> 172.21.15.179:0/3055509222 conn(0x7f7400078240 msgr2=0x7f7400078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:39.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.288+0000 7f7408237700 1 -- 172.21.15.179:0/3055509222 shutdown_connections 2024-04-28T12:41:39.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.288+0000 7f7408237700 1 -- 172.21.15.179:0/3055509222 wait complete. 2024-04-28T12:41:39.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.289+0000 7f7408237700 1 Processor -- start 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.289+0000 7f7408237700 1 -- start start 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.289+0000 7f7408237700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73e80035c0 0x7f740019e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.289+0000 7f7408237700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740019ec80 0x7f7400198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.289+0000 7f7408237700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7400198e40 0x7f74001992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.289+0000 7f7408237700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f740010ae50 con 0x7f740019ec80 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.289+0000 7f7408237700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f740010acd0 con 0x7f7400198e40 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.289+0000 7f7408237700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f740010afd0 con 0x7f73e80035c0 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f7405fd3700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73e80035c0 0x7f740019e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f7405fd3700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73e80035c0 0x7f740019e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:60890/0 (socket says 172.21.15.179:60890) 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f7405fd3700 1 -- 172.21.15.179:0/1820092818 learned_addr learned my addr 172.21.15.179:0/1820092818 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f7405fd3700 1 -- 172.21.15.179:0/1820092818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73e80035c0 msgr2=0x7f740019e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f7405fd3700 1 -- 172.21.15.179:0/1820092818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73e80035c0 msgr2=0x7f740019e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-28T12:41:39.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f7405fd3700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73e80035c0 0x7f740019e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f7405fd3700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73e80035c0 0x7f740019e740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f74067d4700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7400198e40 0x7f74001992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f74057d2700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740019ec80 0x7f7400198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f74067d4700 1 -- 172.21.15.179:0/1820092818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7400198e40 msgr2=0x7f74001992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f74067d4700 1 -- 172.21.15.179:0/1820092818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7400198e40 msgr2=0x7f74001992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f74067d4700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7400198e40 0x7f74001992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.290+0000 7f74067d4700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7400198e40 0x7f74001992f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.291+0000 7f74057d2700 1 -- 172.21.15.179:0/1820092818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73e80035c0 msgr2=0x7f740019e740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.291+0000 7f74057d2700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73e80035c0 0x7f740019e740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.291+0000 7f74057d2700 1 -- 172.21.15.179:0/1820092818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7400198e40 msgr2=0x7f74001992f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.291+0000 7f74057d2700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7400198e40 0x7f74001992f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:39.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.291+0000 7f74057d2700 1 -- 172.21.15.179:0/1820092818 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74001a5610 con 0x7f740019ec80 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.291+0000 7f74057d2700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740019ec80 0x7f7400198900 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f73f400b550 tx=0x7f73f400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.291+0000 7f73feffd700 1 -- 172.21.15.179:0/1820092818 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4004230 con 0x7f740019ec80 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.292+0000 7f7408237700 1 -- 172.21.15.179:0/1820092818 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74001a5840 con 0x7f740019ec80 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.292+0000 7f73feffd700 1 -- 172.21.15.179:0/1820092818 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73f40043d0 con 0x7f740019ec80 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.292+0000 7f73feffd700 1 -- 172.21.15.179:0/1820092818 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4011b90 con 0x7f740019ec80 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.292+0000 7f7408237700 1 -- 172.21.15.179:0/1820092818 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74001a5dd0 con 0x7f740019ec80 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.293+0000 7f7408237700 1 -- 172.21.15.179:0/1820092818 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7400064a80 con 0x7f740019ec80 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.293+0000 7f73feffd700 1 -- 172.21.15.179:0/1820092818 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f73f40036a0 con 0x7f740019ec80 2024-04-28T12:41:39.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.293+0000 7f73feffd700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73ec039ae0 0x7f73ec03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.293+0000 7f73feffd700 1 -- 172.21.15.179:0/1820092818 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f73f404c3a0 con 0x7f740019ec80 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.294+0000 7f7405fd3700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73ec039ae0 0x7f73ec03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:39.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.295+0000 7f7405fd3700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73ec039ae0 0x7f73ec03bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009a30 tx=0x7f73f800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:39.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.308+0000 7f73feffd700 1 -- 172.21.15.179:0/1820092818 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f73f4010380 con 0x7f740019ec80 2024-04-28T12:41:39.592 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.596+0000 7f7408237700 1 -- 172.21.15.179:0/1820092818 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f740002e490 con 0x7f740019ec80 2024-04-28T12:41:39.593 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: mon.c calling monitor election 2024-04-28T12:41:39.593 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:41:39.593 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: mon.a calling monitor election 2024-04-28T12:41:39.593 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:39.593 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:39.593 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:39.593 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:39.593 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: monmap epoch 3 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: last_changed 2024-04-28T12:41:34.197112+0000 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: created 2024-04-28T12:39:18.014667+0000 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: min_mon_release 19 (squid) 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: election_strategy: 1 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-04-28T12:41:39.594 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: 2: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.b 2024-04-28T12:41:39.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: fsmap 2024-04-28T12:41:39.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:41:39.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: mgrmap e13: x(active, since 69s) 2024-04-28T12:41:39.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: overall HEALTH_OK 2024-04-28T12:41:39.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:39.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:39.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:39.595 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:39.595 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.597+0000 7f73feffd700 1 -- 172.21.15.179:0/1820092818 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f73f4003990 con 0x7f740019ec80 2024-04-28T12:41:39.596 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-04-28T12:41:39.596 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":3,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","modified":"2024-04-28T12:41:34.197112Z","created":"2024-04-28T12:39:18.014667Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3300","nonce":0},{"type":"v1","addr":"172.21.15.179:6789","nonce":0}]},"addr":"172.21.15.179:6789/0","public_addr":"172.21.15.179:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-28T12:41:39.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.599+0000 7f7408237700 1 -- 172.21.15.179:0/1820092818 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73ec039ae0 msgr2=0x7f73ec03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:39.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.599+0000 7f7408237700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73ec039ae0 0x7f73ec03bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009a30 tx=0x7f73f800b040 comp rx=0 tx=0).stop 2024-04-28T12:41:39.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.599+0000 7f7408237700 1 -- 172.21.15.179:0/1820092818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740019ec80 msgr2=0x7f7400198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:39.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.599+0000 7f7408237700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740019ec80 0x7f7400198900 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f73f400b550 tx=0x7f73f400b910 comp rx=0 tx=0).stop 2024-04-28T12:41:39.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.599+0000 7f7408237700 1 -- 172.21.15.179:0/1820092818 shutdown_connections 2024-04-28T12:41:39.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.599+0000 7f7408237700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73ec039ae0 0x7f73ec03bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:39.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.599+0000 7f7408237700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7400198e40 0x7f74001992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:39.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.599+0000 7f7408237700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f73e80035c0 0x7f740019e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:39.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.599+0000 7f7408237700 1 --2- 172.21.15.179:0/1820092818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740019ec80 0x7f7400198900 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:39.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.599+0000 7f7408237700 1 -- 172.21.15.179:0/1820092818 >> 172.21.15.179:0/1820092818 conn(0x7f7400078240 msgr2=0x7f74000fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:39.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.599+0000 7f7408237700 1 -- 172.21.15.179:0/1820092818 shutdown_connections 2024-04-28T12:41:39.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:39.599+0000 7f7408237700 1 -- 172.21.15.179:0/1820092818 wait complete. 2024-04-28T12:41:39.598 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 3 2024-04-28T12:41:39.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: mon.c calling monitor election 2024-04-28T12:41:39.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:41:39.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: mon.a calling monitor election 2024-04-28T12:41:39.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:39.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:39.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:39.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:39.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:39.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:39.626 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: monmap epoch 3 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: last_changed 2024-04-28T12:41:34.197112+0000 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: created 2024-04-28T12:39:18.014667+0000 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: min_mon_release 19 (squid) 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: election_strategy: 1 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: 2: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.b 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: fsmap 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:41:39.627 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: mgrmap e13: x(active, since 69s) 2024-04-28T12:41:39.628 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: overall HEALTH_OK 2024-04-28T12:41:39.628 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:39.628 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:39.628 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:39.628 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:40.202 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-28T12:41:40.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-28T12:41:40.590 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:41:40.881 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:40 smithi084 ceph-mon[28081]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:40.881 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:40 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:40.881 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:40 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:40.881 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:40 smithi084 ceph-mon[28081]: from='client.? 172.21.15.179:0/1820092818' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:41:40.881 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:40 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:40 smithi179 ceph-mon[63029]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:40 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:40 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:40 smithi179 ceph-mon[63029]: from='client.? 172.21.15.179:0/1820092818' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:41:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:40 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:41.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: mon.c calling monitor election 2024-04-28T12:41:41.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:41:41.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: mon.a calling monitor election 2024-04-28T12:41:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:41:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-28T12:41:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: monmap epoch 3 2024-04-28T12:41:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: last_changed 2024-04-28T12:41:34.197112+0000 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: created 2024-04-28T12:39:18.014667+0000 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: min_mon_release 19 (squid) 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: election_strategy: 1 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: 2: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.b 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: fsmap 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: mgrmap e13: x(active, since 69s) 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: overall HEALTH_OK 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:41.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:41.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:41.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:41.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:41.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='client.? 172.21.15.179:0/1820092818' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-28T12:41:41.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:42.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: mon.b calling monitor election 2024-04-28T12:41:42.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: Updating smithi084:/etc/ceph/ceph.conf 2024-04-28T12:41:42.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: Updating smithi161:/etc/ceph/ceph.conf 2024-04-28T12:41:42.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: Updating smithi179:/etc/ceph/ceph.conf 2024-04-28T12:41:42.213 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: mon.b calling monitor election 2024-04-28T12:41:42.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: mon.c calling monitor election 2024-04-28T12:41:42.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: mon.a calling monitor election 2024-04-28T12:41:42.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-28T12:41:42.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: monmap epoch 3 2024-04-28T12:41:42.214 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:42.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: last_changed 2024-04-28T12:41:34.197112+0000 2024-04-28T12:41:42.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: created 2024-04-28T12:39:18.014667+0000 2024-04-28T12:41:42.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: min_mon_release 19 (squid) 2024-04-28T12:41:42.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: election_strategy: 1 2024-04-28T12:41:42.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-04-28T12:41:42.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-04-28T12:41:42.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: 2: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.b 2024-04-28T12:41:42.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: fsmap 2024-04-28T12:41:42.215 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:41:42.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: mgrmap e13: x(active, since 71s) 2024-04-28T12:41:42.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: overall HEALTH_OK 2024-04-28T12:41:42.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: Updating smithi161:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:42.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: Updating smithi179:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:42.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:42.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:42.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:42.216 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:42.217 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:42.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: mon.b calling monitor election 2024-04-28T12:41:42.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: Updating smithi084:/etc/ceph/ceph.conf 2024-04-28T12:41:42.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: Updating smithi161:/etc/ceph/ceph.conf 2024-04-28T12:41:42.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: Updating smithi179:/etc/ceph/ceph.conf 2024-04-28T12:41:42.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: mon.b calling monitor election 2024-04-28T12:41:42.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: mon.c calling monitor election 2024-04-28T12:41:42.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: mon.a calling monitor election 2024-04-28T12:41:42.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-28T12:41:42.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: monmap epoch 3 2024-04-28T12:41:42.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:42.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: last_changed 2024-04-28T12:41:34.197112+0000 2024-04-28T12:41:42.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: created 2024-04-28T12:39:18.014667+0000 2024-04-28T12:41:42.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: min_mon_release 19 (squid) 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: election_strategy: 1 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: 2: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.b 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: fsmap 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: mgrmap e13: x(active, since 71s) 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: overall HEALTH_OK 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: Updating smithi161:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: Updating smithi179:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:42.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:42.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: mon.b calling monitor election 2024-04-28T12:41:42.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: Updating smithi084:/etc/ceph/ceph.conf 2024-04-28T12:41:42.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: Updating smithi161:/etc/ceph/ceph.conf 2024-04-28T12:41:42.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: Updating smithi179:/etc/ceph/ceph.conf 2024-04-28T12:41:42.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: mon.b calling monitor election 2024-04-28T12:41:42.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: mon.c calling monitor election 2024-04-28T12:41:42.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: mon.a calling monitor election 2024-04-28T12:41:42.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-28T12:41:42.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: monmap epoch 3 2024-04-28T12:41:42.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:42.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: last_changed 2024-04-28T12:41:34.197112+0000 2024-04-28T12:41:42.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: created 2024-04-28T12:39:18.014667+0000 2024-04-28T12:41:42.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: min_mon_release 19 (squid) 2024-04-28T12:41:42.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: election_strategy: 1 2024-04-28T12:41:42.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-04-28T12:41:42.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: 1: [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon.c 2024-04-28T12:41:42.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: 2: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.b 2024-04-28T12:41:42.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: fsmap 2024-04-28T12:41:42.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: osdmap e4: 0 total, 0 up, 0 in 2024-04-28T12:41:42.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: mgrmap e13: x(active, since 71s) 2024-04-28T12:41:42.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: overall HEALTH_OK 2024-04-28T12:41:42.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: Updating smithi161:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:42.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: Updating smithi179:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:42.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:42.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:42.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:42.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:42.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:42.698 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.702+0000 7f3110c00700 1 -- 172.21.15.84:0/3013982950 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f310c108ec0 msgr2=0x7f310c1092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:42.698 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.702+0000 7f3110c00700 1 --2- 172.21.15.84:0/3013982950 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f310c108ec0 0x7f310c1092a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009910 tx=0x7f30fc009c20 comp rx=0 tx=0).stop 2024-04-28T12:41:42.699 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.702+0000 7f3110c00700 1 -- 172.21.15.84:0/3013982950 shutdown_connections 2024-04-28T12:41:42.699 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.702+0000 7f3110c00700 1 --2- 172.21.15.84:0/3013982950 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f310c108ec0 0x7f310c1092a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:42.699 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.702+0000 7f3110c00700 1 -- 172.21.15.84:0/3013982950 >> 172.21.15.84:0/3013982950 conn(0x7f310c077880 msgr2=0x7f310c077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:42.699 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.702+0000 7f3110c00700 1 -- 172.21.15.84:0/3013982950 shutdown_connections 2024-04-28T12:41:42.699 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.702+0000 7f3110c00700 1 -- 172.21.15.84:0/3013982950 wait complete. 2024-04-28T12:41:42.699 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.704+0000 7f3110c00700 1 Processor -- start 2024-04-28T12:41:42.700 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.704+0000 7f3110c00700 1 -- start start 2024-04-28T12:41:42.700 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.704+0000 7f3110c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f310c108ec0 0x7f310c19e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:42.700 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.704+0000 7f3110c00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f310c19ec80 0x7f310c198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:42.700 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.704+0000 7f3110c00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f310c198e40 0x7f310c1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:42.700 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.704+0000 7f3110c00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f310c10d300 con 0x7f310c198e40 2024-04-28T12:41:42.701 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.704+0000 7f3110c00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f310c10d180 con 0x7f310c19ec80 2024-04-28T12:41:42.701 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.704+0000 7f3110c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f310c10d480 con 0x7f310c108ec0 2024-04-28T12:41:42.702 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.705+0000 7f3109e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f310c108ec0 0x7f310c19e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:42.702 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.705+0000 7f310a800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f310c198e40 0x7f310c1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:42.702 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.705+0000 7f3109e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f310c108ec0 0x7f310c19e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:37424/0 (socket says 172.21.15.84:37424) 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.705+0000 7f3109e00700 1 -- 172.21.15.84:0/902241280 learned_addr learned my addr 172.21.15.84:0/902241280 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.705+0000 7f3109400700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f310c19ec80 0x7f310c198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.705+0000 7f3109e00700 1 -- 172.21.15.84:0/902241280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f310c19ec80 msgr2=0x7f310c198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.705+0000 7f3109e00700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f310c19ec80 0x7f310c198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.705+0000 7f3109e00700 1 -- 172.21.15.84:0/902241280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f310c198e40 msgr2=0x7f310c1992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.705+0000 7f3109e00700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f310c198e40 0x7f310c1992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.705+0000 7f3109e00700 1 -- 172.21.15.84:0/902241280 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310c1a5610 con 0x7f310c108ec0 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.705+0000 7f310a800700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f310c198e40 0x7f310c1992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.706+0000 7f3109e00700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f310c108ec0 0x7f310c19e740 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f30fc005b40 tx=0x7f30fc004f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.706+0000 7f3102a00700 1 -- 172.21.15.84:0/902241280 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc026070 con 0x7f310c108ec0 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.706+0000 7f3102a00700 1 -- 172.21.15.84:0/902241280 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30fc00b6f0 con 0x7f310c108ec0 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.706+0000 7f3102a00700 1 -- 172.21.15.84:0/902241280 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc009e60 con 0x7f310c108ec0 2024-04-28T12:41:42.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.706+0000 7f3110c00700 1 -- 172.21.15.84:0/902241280 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f310c1a5840 con 0x7f310c108ec0 2024-04-28T12:41:42.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.706+0000 7f3110c00700 1 -- 172.21.15.84:0/902241280 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f310c1a5d30 con 0x7f310c108ec0 2024-04-28T12:41:42.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.707+0000 7f3102a00700 1 -- 172.21.15.84:0/902241280 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f30fc029070 con 0x7f310c108ec0 2024-04-28T12:41:42.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.707+0000 7f3110c00700 1 -- 172.21.15.84:0/902241280 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f310c064a80 con 0x7f310c108ec0 2024-04-28T12:41:42.705 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.708+0000 7f3102a00700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f30f0039ae0 0x7f30f003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:42.705 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.708+0000 7f3102a00700 1 -- 172.21.15.84:0/902241280 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f30fc04d4a0 con 0x7f310c108ec0 2024-04-28T12:41:42.705 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.708+0000 7f3109400700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f30f0039ae0 0x7f30f003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:42.705 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.709+0000 7f3109400700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f30f0039ae0 0x7f30f003bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009ad0 tx=0x7f30f8009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:42.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.713+0000 7f3102a00700 1 -- 172.21.15.84:0/902241280 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f30fc013070 con 0x7f310c108ec0 2024-04-28T12:41:42.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.946+0000 7f3110c00700 1 -- 172.21.15.84:0/902241280 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f310c1a5ff0 con 0x7f310c108ec0 2024-04-28T12:41:42.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.947+0000 7f3102a00700 1 -- 172.21.15.84:0/902241280 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f30fc030130 con 0x7f310c108ec0 2024-04-28T12:41:42.943 INFO:teuthology.orchestra.run.smithi084.stdout:# minimal ceph.conf for ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:42.943 INFO:teuthology.orchestra.run.smithi084.stdout:[global] 2024-04-28T12:41:42.943 INFO:teuthology.orchestra.run.smithi084.stdout: fsid = ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:41:42.943 INFO:teuthology.orchestra.run.smithi084.stdout: mon_host = [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] 2024-04-28T12:41:42.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.949+0000 7f3110c00700 1 -- 172.21.15.84:0/902241280 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f30f0039ae0 msgr2=0x7f30f003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:42.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.949+0000 7f3110c00700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f30f0039ae0 0x7f30f003bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009ad0 tx=0x7f30f8009210 comp rx=0 tx=0).stop 2024-04-28T12:41:42.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.949+0000 7f3110c00700 1 -- 172.21.15.84:0/902241280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f310c108ec0 msgr2=0x7f310c19e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:42.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.949+0000 7f3110c00700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f310c108ec0 0x7f310c19e740 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f30fc005b40 tx=0x7f30fc004f80 comp rx=0 tx=0).stop 2024-04-28T12:41:42.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.950+0000 7f3110c00700 1 -- 172.21.15.84:0/902241280 shutdown_connections 2024-04-28T12:41:42.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.950+0000 7f3110c00700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f310c198e40 0x7f310c1992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:42.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.950+0000 7f3110c00700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f310c108ec0 0x7f310c19e740 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:42.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.950+0000 7f3110c00700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f30f0039ae0 0x7f30f003bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:42.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.950+0000 7f3110c00700 1 --2- 172.21.15.84:0/902241280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f310c19ec80 0x7f310c198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:42.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.950+0000 7f3110c00700 1 -- 172.21.15.84:0/902241280 >> 172.21.15.84:0/902241280 conn(0x7f310c077880 msgr2=0x7f310c0ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:42.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.950+0000 7f3110c00700 1 -- 172.21.15.84:0/902241280 shutdown_connections 2024-04-28T12:41:42.947 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:42.950+0000 7f3110c00700 1 -- 172.21.15.84:0/902241280 wait complete. 2024-04-28T12:41:43.194 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: Updating smithi084:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:43.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:43.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:43.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:41:43.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:43.195 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/902241280' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:43.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: Updating smithi084:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:43.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:43.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:43.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:41:43.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:43.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:43 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/902241280' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:43.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: Updating smithi084:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-28T12:41:43.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:41:43.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:43.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:43.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:41:43.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:43.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:43 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/902241280' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:43.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:41:43 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:41:43.197+0000 7f01e2000700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-28T12:41:43.720 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-28T12:41:43.720 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:41:43.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.conf 2024-04-28T12:41:43.803 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:41:43.804 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:43.925 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-28T12:41:43.925 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.conf 2024-04-28T12:41:43.957 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-28T12:41:43.957 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:44.025 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-28T12:41:44.025 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.conf 2024-04-28T12:41:44.052 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-28T12:41:44.052 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-28T12:41:44.116 INFO:tasks.cephadm:Adding mgr.x on smithi084 2024-04-28T12:41:44.117 INFO:tasks.cephadm:Adding mgr.y on smithi161 2024-04-28T12:41:44.117 INFO:tasks.cephadm:Adding mgr.z on smithi179 2024-04-28T12:41:44.118 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi084=x;smithi161=y;smithi179=z' 2024-04-28T12:41:44.175 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:44 smithi084 ceph-mon[28081]: Reconfiguring mon.a (unknown last config time)... 2024-04-28T12:41:44.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:44 smithi084 ceph-mon[28081]: Reconfiguring daemon mon.a on smithi084 2024-04-28T12:41:44.176 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:44 smithi084 ceph-mon[28081]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:44.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:44 smithi161 ceph-mon[62387]: Reconfiguring mon.a (unknown last config time)... 2024-04-28T12:41:44.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:44 smithi161 ceph-mon[62387]: Reconfiguring daemon mon.a on smithi084 2024-04-28T12:41:44.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:44 smithi161 ceph-mon[62387]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:44.382 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:41:44.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:44 smithi179 ceph-mon[63029]: Reconfiguring mon.a (unknown last config time)... 2024-04-28T12:41:44.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:44 smithi179 ceph-mon[63029]: Reconfiguring daemon mon.a on smithi084 2024-04-28T12:41:44.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:44 smithi179 ceph-mon[63029]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:45.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.817+0000 7fccbb43b700 1 -- 172.21.15.179:0/3340318298 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccb4101840 msgr2=0x7fccb4103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:45.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.817+0000 7fccbb43b700 1 --2- 172.21.15.179:0/3340318298 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccb4101840 0x7fccb4103c70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009910 tx=0x7fccb0009c20 comp rx=0 tx=0).stop 2024-04-28T12:41:45.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.817+0000 7fccbb43b700 1 -- 172.21.15.179:0/3340318298 shutdown_connections 2024-04-28T12:41:45.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.817+0000 7fccbb43b700 1 --2- 172.21.15.179:0/3340318298 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccb4101840 0x7fccb4103c70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:45.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.817+0000 7fccbb43b700 1 -- 172.21.15.179:0/3340318298 >> 172.21.15.179:0/3340318298 conn(0x7fccb40fd460 msgr2=0x7fccb40ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:45.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.818+0000 7fccbb43b700 1 -- 172.21.15.179:0/3340318298 shutdown_connections 2024-04-28T12:41:45.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.818+0000 7fccbb43b700 1 -- 172.21.15.179:0/3340318298 wait complete. 2024-04-28T12:41:45.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.818+0000 7fccbb43b700 1 Processor -- start 2024-04-28T12:41:45.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.818+0000 7fccbb43b700 1 -- start start 2024-04-28T12:41:45.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.818+0000 7fccbb43b700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccb4101840 0x7fccb419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.818+0000 7fccbb43b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccb419f280 0x7fccb41a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.818+0000 7fccbb43b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccb419f700 0x7fccb41a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.818+0000 7fccbb43b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fccb4106db0 con 0x7fccb419f700 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccbb43b700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fccb4106c30 con 0x7fccb419f280 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccbb43b700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fccb4106f30 con 0x7fccb4101840 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccb89d6700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccb419f280 0x7fccb41a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccb91d7700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccb4101840 0x7fccb419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccb89d6700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccb419f280 0x7fccb41a42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.179:48102/0 (socket says 172.21.15.179:48102) 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccb89d6700 1 -- 172.21.15.179:0/1808910109 learned_addr learned my addr 172.21.15.179:0/1808910109 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccb91d7700 1 -- 172.21.15.179:0/1808910109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccb419f280 msgr2=0x7fccb41a42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccb91d7700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccb419f280 0x7fccb41a42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccb91d7700 1 -- 172.21.15.179:0/1808910109 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccb419f700 msgr2=0x7fccb41a6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccb91d7700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccb419f700 0x7fccb41a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:45.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccb91d7700 1 -- 172.21.15.179:0/1808910109 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb41a7160 con 0x7fccb4101840 2024-04-28T12:41:45.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccb89d6700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccb419f280 0x7fccb41a42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:41:45.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccb91d7700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccb4101840 0x7fccb419ed40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fccb0000c00 tx=0x7fccb0005020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:45.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccaa7fc700 1 -- 172.21.15.179:0/1808910109 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0026070 con 0x7fccb4101840 2024-04-28T12:41:45.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccaa7fc700 1 -- 172.21.15.179:0/1808910109 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccb000b750 con 0x7fccb4101840 2024-04-28T12:41:45.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccbb43b700 1 -- 172.21.15.179:0/1808910109 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccb41a73f0 con 0x7fccb4101840 2024-04-28T12:41:45.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccaa7fc700 1 -- 172.21.15.179:0/1808910109 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb00038e0 con 0x7fccb4101840 2024-04-28T12:41:45.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.819+0000 7fccbb43b700 1 -- 172.21.15.179:0/1808910109 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccb41a7950 con 0x7fccb4101840 2024-04-28T12:41:45.819 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.820+0000 7fccaa7fc700 1 -- 172.21.15.179:0/1808910109 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fccb0029070 con 0x7fccb4101840 2024-04-28T12:41:45.819 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.820+0000 7fccbb43b700 1 -- 172.21.15.179:0/1808910109 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccb4064a80 con 0x7fccb4101840 2024-04-28T12:41:45.819 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.821+0000 7fccaa7fc700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcca0039ae0 0x7fcca003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:45.819 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.821+0000 7fccaa7fc700 1 -- 172.21.15.179:0/1808910109 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fccb004d590 con 0x7fccb4101840 2024-04-28T12:41:45.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.821+0000 7fccb89d6700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcca0039ae0 0x7fcca003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:45.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.822+0000 7fccb89d6700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcca0039ae0 0x7fcca003bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009f70 tx=0x7fcca40092c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:45.822 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:45.826+0000 7fccaa7fc700 1 -- 172.21.15.179:0/1808910109 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fccb0013070 con 0x7fccb4101840 2024-04-28T12:41:46.030 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.033+0000 7fccbb43b700 1 -- 172.21.15.179:0/1808910109 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi084=x;smithi161=y;smithi179=z", "target": ["mon-mgr", ""]}) -- 0x7fccb40638f0 con 0x7fcca0039ae0 2024-04-28T12:41:46.044 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.048+0000 7fccaa7fc700 1 -- 172.21.15.179:0/1808910109 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fccb40638f0 con 0x7fcca0039ae0 2024-04-28T12:41:46.045 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mgr update... 2024-04-28T12:41:46.047 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.050+0000 7fccbb43b700 1 -- 172.21.15.179:0/1808910109 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcca0039ae0 msgr2=0x7fcca003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:46.047 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.050+0000 7fccbb43b700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcca0039ae0 0x7fcca003bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcca4009f70 tx=0x7fcca40092c0 comp rx=0 tx=0).stop 2024-04-28T12:41:46.047 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.050+0000 7fccbb43b700 1 -- 172.21.15.179:0/1808910109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccb4101840 msgr2=0x7fccb419ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:46.047 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.050+0000 7fccbb43b700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccb4101840 0x7fccb419ed40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fccb0000c00 tx=0x7fccb0005020 comp rx=0 tx=0).stop 2024-04-28T12:41:46.047 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.050+0000 7fccbb43b700 1 -- 172.21.15.179:0/1808910109 shutdown_connections 2024-04-28T12:41:46.047 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.050+0000 7fccbb43b700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccb419f700 0x7fccb41a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:46.048 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.050+0000 7fccbb43b700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccb4101840 0x7fccb419ed40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:46.048 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.050+0000 7fccbb43b700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcca0039ae0 0x7fcca003bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:46.048 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.050+0000 7fccbb43b700 1 --2- 172.21.15.179:0/1808910109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccb419f280 0x7fccb41a42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:46.048 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.050+0000 7fccbb43b700 1 -- 172.21.15.179:0/1808910109 >> 172.21.15.179:0/1808910109 conn(0x7fccb40fd460 msgr2=0x7fccb40fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:46.048 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.050+0000 7fccbb43b700 1 -- 172.21.15.179:0/1808910109 shutdown_connections 2024-04-28T12:41:46.048 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:41:46.050+0000 7fccbb43b700 1 -- 172.21.15.179:0/1808910109 wait complete. 2024-04-28T12:41:46.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:46.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:46.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:46 smithi179 ceph-mon[63029]: Reconfiguring mon.b (monmap changed)... 2024-04-28T12:41:46.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:46.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:41:46.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:46.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:46 smithi179 ceph-mon[63029]: Reconfiguring daemon mon.b on smithi161 2024-04-28T12:41:46.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:46 smithi179 ceph-mon[63029]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:46.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:46.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:46.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:46.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:46 smithi084 ceph-mon[28081]: Reconfiguring mon.b (monmap changed)... 2024-04-28T12:41:46.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:46.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:41:46.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:46.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:46 smithi084 ceph-mon[28081]: Reconfiguring daemon mon.b on smithi161 2024-04-28T12:41:46.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:46 smithi084 ceph-mon[28081]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:46.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:46.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:46.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:46.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:46 smithi161 ceph-mon[62387]: Reconfiguring mon.b (monmap changed)... 2024-04-28T12:41:46.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:46.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:41:46.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:46.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:46 smithi161 ceph-mon[62387]: Reconfiguring daemon mon.b on smithi161 2024-04-28T12:41:46.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:46 smithi161 ceph-mon[62387]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:46.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:46.629 DEBUG:teuthology.orchestra.run.smithi161:mgr.y> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.y.service 2024-04-28T12:41:46.632 DEBUG:teuthology.orchestra.run.smithi179:mgr.z> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.z.service 2024-04-28T12:41:46.634 INFO:tasks.cephadm:Deploying OSDs... 2024-04-28T12:41:46.635 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:41:46.635 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-04-28T12:41:46.692 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-28T12:41:46.693 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 2024-04-28T12:41:46.743 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-28T12:41:46.744 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:41:46.744 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-04-28T12:41:46.744 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:41:46.744 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:41:46.744 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-04-28 12:40:42.543378254 +0000 2024-04-28T12:41:46.744 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-04-28 12:30:50.230008893 +0000 2024-04-28T12:41:46.744 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-04-28 12:30:50.230008893 +0000 2024-04-28T12:41:46.744 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-04-28 12:30:50.230008893 +0000 2024-04-28T12:41:46.745 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-28T12:41:46.821 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-04-28T12:41:46.822 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-04-28T12:41:46.822 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000390273 s, 1.3 MB/s 2024-04-28T12:41:46.826 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-28T12:41:46.882 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 2024-04-28T12:41:46.934 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-28T12:41:46.934 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:41:46.934 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-04-28T12:41:46.934 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:41:46.934 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:41:46.935 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-04-28 12:40:42.601377260 +0000 2024-04-28T12:41:46.935 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-04-28 12:30:50.832998886 +0000 2024-04-28T12:41:46.935 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-04-28 12:30:50.832998886 +0000 2024-04-28T12:41:46.935 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-04-28 12:30:50.832998886 +0000 2024-04-28T12:41:46.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-28T12:41:47.013 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-04-28T12:41:47.013 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-04-28T12:41:47.013 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000375455 s, 1.4 MB/s 2024-04-28T12:41:47.017 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-28T12:41:47.071 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 2024-04-28T12:41:47.118 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-28T12:41:47.118 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:41:47.119 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-04-28T12:41:47.119 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:41:47.119 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:41:47.119 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-04-28 12:40:42.663376197 +0000 2024-04-28T12:41:47.119 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-04-28 12:30:51.451988613 +0000 2024-04-28T12:41:47.119 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-04-28 12:30:51.451988613 +0000 2024-04-28T12:41:47.119 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-04-28 12:30:51.451988613 +0000 2024-04-28T12:41:47.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-28T12:41:47.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:47 smithi084 ceph-mon[28081]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi084=x;smithi161=y;smithi179=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:47 smithi084 ceph-mon[28081]: Saving service mgr spec with placement smithi084=x;smithi161=y;smithi179=z;count:3 2024-04-28T12:41:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:41:47.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:47.245 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-04-28T12:41:47.245 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-04-28T12:41:47.245 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000405212 s, 1.3 MB/s 2024-04-28T12:41:47.249 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-28T12:41:47.312 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:47 smithi179 ceph-mon[63029]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi084=x;smithi161=y;smithi179=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:47.312 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:47 smithi179 ceph-mon[63029]: Saving service mgr spec with placement smithi084=x;smithi161=y;smithi179=z;count:3 2024-04-28T12:41:47.312 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:47.312 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:47.312 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:47.312 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:41:47.313 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:47.350 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 2024-04-28T12:41:47.403 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-28T12:41:47.403 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:41:47.403 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-04-28T12:41:47.403 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:41:47.404 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:41:47.404 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-04-28 12:40:42.713375341 +0000 2024-04-28T12:41:47.404 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-04-28 12:30:52.025979088 +0000 2024-04-28T12:41:47.404 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-04-28 12:30:52.025979088 +0000 2024-04-28T12:41:47.404 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-04-28 12:30:52.025979088 +0000 2024-04-28T12:41:47.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-28T12:41:47.486 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-04-28T12:41:47.486 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-04-28T12:41:47.486 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000357769 s, 1.4 MB/s 2024-04-28T12:41:47.490 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-28T12:41:47.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:47 smithi161 ceph-mon[62387]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi084=x;smithi161=y;smithi179=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:47.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:47 smithi161 ceph-mon[62387]: Saving service mgr spec with placement smithi084=x;smithi161=y;smithi179=z;count:3 2024-04-28T12:41:47.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:47.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:47.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-28T12:41:47.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-28T12:41:47.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:47.542 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-28T12:41:47.542 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2024-04-28T12:41:47.561 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-28T12:41:47.562 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 2024-04-28T12:41:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-28T12:41:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:41:47.619 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-28T12:41:47.619 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:41:47.619 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:41:47.619 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-04-28 12:41:33.605762783 +0000 2024-04-28T12:41:47.619 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-04-28 12:29:55.311760025 +0000 2024-04-28T12:41:47.619 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-04-28 12:29:55.311760025 +0000 2024-04-28T12:41:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-04-28 12:29:55.311760025 +0000 2024-04-28T12:41:47.620 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-28T12:41:47.687 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-04-28T12:41:47.687 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-04-28T12:41:47.687 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.0002879 s, 1.8 MB/s 2024-04-28T12:41:47.688 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-28T12:41:47.747 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 2024-04-28T12:41:47.804 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-28T12:41:47.805 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:41:47.805 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-28T12:41:47.805 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:41:47.805 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:41:47.805 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-04-28 12:41:33.605762783 +0000 2024-04-28T12:41:47.805 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-04-28 12:29:55.666752433 +0000 2024-04-28T12:41:47.805 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-04-28 12:29:55.666752433 +0000 2024-04-28T12:41:47.805 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-04-28 12:29:55.666752433 +0000 2024-04-28T12:41:47.805 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-28T12:41:47.870 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-04-28T12:41:47.871 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-04-28T12:41:47.871 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000204971 s, 2.5 MB/s 2024-04-28T12:41:47.872 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-28T12:41:47.929 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 2024-04-28T12:41:47.986 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-28T12:41:47.986 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:41:47.986 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-28T12:41:47.986 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:41:47.986 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:41:47.986 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-04-28 12:41:33.606762763 +0000 2024-04-28T12:41:47.986 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-04-28 12:29:56.011745055 +0000 2024-04-28T12:41:47.986 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-04-28 12:29:56.011745055 +0000 2024-04-28T12:41:47.986 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-04-28 12:29:56.011745055 +0000 2024-04-28T12:41:47.987 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-28T12:41:48.051 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-04-28T12:41:48.051 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-04-28T12:41:48.051 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000230968 s, 2.2 MB/s 2024-04-28T12:41:48.052 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-28T12:41:48.109 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 2024-04-28T12:41:48.165 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-28T12:41:48.165 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:41:48.165 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-28T12:41:48.165 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:41:48.165 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:41:48.166 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-04-28 12:41:33.606762763 +0000 2024-04-28T12:41:48.166 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-04-28 12:29:56.352737763 +0000 2024-04-28T12:41:48.166 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-04-28 12:29:56.352737763 +0000 2024-04-28T12:41:48.166 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-04-28 12:29:56.352737763 +0000 2024-04-28T12:41:48.166 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-28T12:41:48.229 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-04-28T12:41:48.229 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-04-28T12:41:48.229 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.00026115 s, 2.0 MB/s 2024-04-28T12:41:48.230 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-28T12:41:48.287 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-04-28T12:41:48.287 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2024-04-28T12:41:48.303 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-28T12:41:48.303 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 2024-04-28T12:41:48.358 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-28T12:41:48.358 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:41:48.358 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-28T12:41:48.358 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:41:48.358 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:41:48.358 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-04-28 12:41:23.887786645 +0000 2024-04-28T12:41:48.358 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-04-28 12:29:47.708628807 +0000 2024-04-28T12:41:48.358 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-04-28 12:29:47.708628807 +0000 2024-04-28T12:41:48.358 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-04-28 12:29:47.708628807 +0000 2024-04-28T12:41:48.359 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-28T12:41:48.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:48 smithi179 ceph-mon[63029]: Reconfiguring mon.c (monmap changed)... 2024-04-28T12:41:48.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:48 smithi179 ceph-mon[63029]: Reconfiguring daemon mon.c on smithi179 2024-04-28T12:41:48.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:48 smithi179 ceph-mon[63029]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:48.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:48 smithi084 ceph-mon[28081]: Reconfiguring mon.c (monmap changed)... 2024-04-28T12:41:48.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:48 smithi084 ceph-mon[28081]: Reconfiguring daemon mon.c on smithi179 2024-04-28T12:41:48.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:48 smithi084 ceph-mon[28081]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:48.421 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-04-28T12:41:48.421 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-04-28T12:41:48.421 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000216104 s, 2.4 MB/s 2024-04-28T12:41:48.422 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-28T12:41:48.479 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 2024-04-28T12:41:48.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:48 smithi161 ceph-mon[62387]: Reconfiguring mon.c (monmap changed)... 2024-04-28T12:41:48.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:48 smithi161 ceph-mon[62387]: Reconfiguring daemon mon.c on smithi179 2024-04-28T12:41:48.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:48 smithi161 ceph-mon[62387]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:48.534 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-28T12:41:48.534 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:41:48.534 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-28T12:41:48.534 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:41:48.534 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:41:48.534 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-04-28 12:41:23.888786624 +0000 2024-04-28T12:41:48.534 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-04-28 12:29:48.064620852 +0000 2024-04-28T12:41:48.534 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-04-28 12:29:48.064620852 +0000 2024-04-28T12:41:48.534 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-04-28 12:29:48.064620852 +0000 2024-04-28T12:41:48.535 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-28T12:41:48.599 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-04-28T12:41:48.599 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-04-28T12:41:48.599 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000225025 s, 2.3 MB/s 2024-04-28T12:41:48.600 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-28T12:41:48.656 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 2024-04-28T12:41:48.710 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-28T12:41:48.711 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:41:48.711 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-28T12:41:48.711 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:41:48.711 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:41:48.711 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-04-28 12:41:23.888786624 +0000 2024-04-28T12:41:48.711 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-04-28 12:29:48.403613277 +0000 2024-04-28T12:41:48.711 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-04-28 12:29:48.403613277 +0000 2024-04-28T12:41:48.711 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-04-28 12:29:48.403613277 +0000 2024-04-28T12:41:48.711 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-28T12:41:48.773 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-04-28T12:41:48.774 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-04-28T12:41:48.774 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000213076 s, 2.4 MB/s 2024-04-28T12:41:48.775 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-28T12:41:48.831 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 2024-04-28T12:41:48.886 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-28T12:41:48.886 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-28T12:41:48.886 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-28T12:41:48.886 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-28T12:41:48.886 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-04-28T12:41:48.886 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-04-28 12:41:23.888786624 +0000 2024-04-28T12:41:48.886 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-04-28 12:29:48.714606328 +0000 2024-04-28T12:41:48.886 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-04-28 12:29:48.714606328 +0000 2024-04-28T12:41:48.886 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-04-28 12:29:48.714606328 +0000 2024-04-28T12:41:48.887 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-28T12:41:48.948 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-04-28T12:41:48.948 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-04-28T12:41:48.948 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000215851 s, 2.4 MB/s 2024-04-28T12:41:48.949 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-28T12:41:49.004 INFO:tasks.cephadm:Deploying osd.0 on smithi084 with /dev/vg_nvme/lv_4... 2024-04-28T12:41:49.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-28T12:41:49.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:49.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:49.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:49.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:49.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:49.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:49.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:41:49.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T12:41:49.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:41:49.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:49.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:49 smithi179 ceph-mon[63029]: Deploying daemon mgr.z on smithi179 2024-04-28T12:41:49.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:49.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:49.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:49.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:49.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:49.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:49.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:41:49.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T12:41:49.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:41:49.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:49.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:49 smithi084 ceph-mon[28081]: Deploying daemon mgr.z on smithi179 2024-04-28T12:41:49.437 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:41:49.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:49.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:49.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:49.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:49.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:49.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:49.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:41:49.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T12:41:49.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:41:49.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:49.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:49 smithi161 ceph-mon[62387]: Deploying daemon mgr.z on smithi179 2024-04-28T12:41:50.370 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:50 smithi179 ceph-mon[63029]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:50.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:50 smithi161 ceph-mon[62387]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:50.606 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:50 smithi084 ceph-mon[28081]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:50.655 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:50 smithi179 ceph-mgr[64895]: -- 172.21.15.179:0/1093564741 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ae7fba45a0 con 0x55ae7fb78800 2024-04-28T12:41:50.915 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:50 smithi179 bash[64876]: 5fab2ce76a48d607b8d43f78d686d3fd64d4ac335b2cecbdea2c6a86668a83b2 2024-04-28T12:41:50.915 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:50 smithi179 systemd[1]: Started Ceph mgr.z for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:41:50.915 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:50 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:50.756+0000 7fafae308200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T12:41:51.346 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:50 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:50.941+0000 7fafae308200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:41:51.346 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:51 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:51.043+0000 7fafae308200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T12:41:51.346 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:51 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:51.349+0000 7fafae308200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T12:41:51.665 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:51 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:51.433+0000 7fafae308200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T12:41:51.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:51 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:51 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:51 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:51 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:41:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:51 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T12:41:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:51 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:41:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:51 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:51 smithi161 ceph-mon[62387]: Deploying daemon mgr.y on smithi161 2024-04-28T12:41:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:51 smithi161 ceph-mon[62387]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:51 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:51 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:51 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:51 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:41:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:51 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T12:41:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:51 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:41:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:51 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:52.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:51 smithi084 ceph-mon[28081]: Deploying daemon mgr.y on smithi161 2024-04-28T12:41:52.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:51 smithi084 ceph-mon[28081]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:52.161 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-28T12:41:52.161 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-28T12:41:52.162 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10+0 records in 2024-04-28T12:41:52.162 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2024-04-28T12:41:52.162 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0318719 s, 329 MB/s 2024-04-28T12:41:52.162 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping successful for: 2024-04-28T12:41:52.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:51 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:52.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:51 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:52.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:51 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:52.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:51 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:41:52.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:51 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-28T12:41:52.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:51 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:41:52.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:51 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:52.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:51 smithi179 ceph-mon[63029]: Deploying daemon mgr.y on smithi161 2024-04-28T12:41:52.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:51 smithi179 ceph-mon[63029]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:52.883 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi084:vg_nvme/lv_4 2024-04-28T12:41:52.915 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:52 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:52.629+0000 7fafae308200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T12:41:52.915 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:52 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:52.799+0000 7fafae308200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T12:41:53.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:53 smithi161 systemd[1]: Starting Ceph mgr.y for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:41:53.251 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:52 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:52.971+0000 7fafae308200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T12:41:53.251 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:53 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:53.255+0000 7fafae308200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T12:41:53.305 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:41:53.547 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:53 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:53.347+0000 7fafae308200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T12:41:53.547 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:53 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:53.550+0000 7fafae308200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T12:41:53.583 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:53 smithi161 podman[63750]: 2024-04-28 12:41:53.176575139 +0000 UTC m=+0.017680345 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:41:53.583 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:53 smithi161 podman[63750]: 2024-04-28 12:41:53.293603991 +0000 UTC m=+0.134709196 container create 5cbde99f508802f5db752cf30af795ad1b90e31c87f349c76aaaa6678c69b24c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-28T12:41:53.583 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:53 smithi161 podman[63750]: 2024-04-28 12:41:53.504363341 +0000 UTC m=+0.345468546 container init 5cbde99f508802f5db752cf30af795ad1b90e31c87f349c76aaaa6678c69b24c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T12:41:53.583 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:53 smithi161 podman[63750]: 2024-04-28 12:41:53.506741115 +0000 UTC m=+0.347846326 container start 5cbde99f508802f5db752cf30af795ad1b90e31c87f349c76aaaa6678c69b24c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T12:41:53.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:53 smithi161 bash[63750]: 5cbde99f508802f5db752cf30af795ad1b90e31c87f349c76aaaa6678c69b24c 2024-04-28T12:41:53.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:53 smithi161 systemd[1]: Started Ceph mgr.y for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:41:53.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:53 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:53.666+0000 7f1d79ccf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-28T12:41:53.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:53 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:53.853+0000 7f1d79ccf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:41:53.915 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:53 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:53.741+0000 7fafae308200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:41:54.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:53 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:53.958+0000 7f1d79ccf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-28T12:41:54.637 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:54 smithi084 ceph-mon[28081]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:54.637 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:54 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:54.637 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:54 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:54.638 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:54 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:54.638 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:54 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:54.638 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:54 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:54.665 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:54 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:54.360+0000 7fafae308200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T12:41:54.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:54 smithi179 ceph-mon[63029]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:54.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:54 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:54.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:54 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:54.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:54 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:54.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:54 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:54.666 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:54 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:54.742 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:54 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:54.289+0000 7f1d79ccf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-28T12:41:54.743 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:54 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:54.376+0000 7f1d79ccf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-28T12:41:54.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:54 smithi161 ceph-mon[62387]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:54.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:54 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:54.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:54 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:54.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:54 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:54.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:54 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:54.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:54 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:55.057 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:54 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:54.804+0000 7fafae308200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T12:41:55.057 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:54 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:54.905+0000 7fafae308200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T12:41:55.415 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:55 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:55.109+0000 7fafae308200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T12:41:55.418 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.422+0000 7f57f0a00700 1 -- 172.21.15.84:0/4033601141 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57ec103580 msgr2=0x7f57ec1039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:55.418 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.422+0000 7f57f0a00700 1 --2- 172.21.15.84:0/4033601141 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57ec103580 0x7f57ec1039e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57d80099c0 tx=0x7f57d8009cd0 comp rx=0 tx=0).stop 2024-04-28T12:41:55.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.422+0000 7f57f0a00700 1 -- 172.21.15.84:0/4033601141 shutdown_connections 2024-04-28T12:41:55.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.422+0000 7f57f0a00700 1 --2- 172.21.15.84:0/4033601141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec103f20 0x7f57ec10a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:55.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.422+0000 7f57f0a00700 1 --2- 172.21.15.84:0/4033601141 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57ec102380 0x7f57ec1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:55.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.422+0000 7f57f0a00700 1 --2- 172.21.15.84:0/4033601141 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57ec103580 0x7f57ec1039e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:55.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.422+0000 7f57f0a00700 1 -- 172.21.15.84:0/4033601141 >> 172.21.15.84:0/4033601141 conn(0x7f57ec0fd920 msgr2=0x7f57ec0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:41:55.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.422+0000 7f57f0a00700 1 -- 172.21.15.84:0/4033601141 shutdown_connections 2024-04-28T12:41:55.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.422+0000 7f57f0a00700 1 -- 172.21.15.84:0/4033601141 wait complete. 2024-04-28T12:41:55.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.423+0000 7f57f0a00700 1 Processor -- start 2024-04-28T12:41:55.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.424+0000 7f57f0a00700 1 -- start start 2024-04-28T12:41:55.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.424+0000 7f57f0a00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec102380 0x7f57ec199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:55.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.424+0000 7f57f0a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57ec103580 0x7f57ec199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:55.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.424+0000 7f57f0a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57ec103f20 0x7f57ec1a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:55.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.424+0000 7f57f0a00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f57ec10ddf0 con 0x7f57ec103580 2024-04-28T12:41:55.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.424+0000 7f57f0a00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f57ec10dc70 con 0x7f57ec102380 2024-04-28T12:41:55.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.424+0000 7f57f0a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f57ec10df70 con 0x7f57ec103f20 2024-04-28T12:41:55.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.424+0000 7f57e9e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec102380 0x7f57ec199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:55.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.424+0000 7f57ea800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57ec103f20 0x7f57ec1a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:55.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.424+0000 7f57e9e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec102380 0x7f57ec199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:49598/0 (socket says 172.21.15.84:49598) 2024-04-28T12:41:55.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.424+0000 7f57e9e00700 1 -- 172.21.15.84:0/1906846250 learned_addr learned my addr 172.21.15.84:0/1906846250 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:41:55.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.424+0000 7f57e9400700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57ec103580 0x7f57ec199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:55.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.425+0000 7f57e9e00700 1 -- 172.21.15.84:0/1906846250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57ec103f20 msgr2=0x7f57ec1a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:55.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.425+0000 7f57e9e00700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57ec103f20 0x7f57ec1a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:55.422 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.425+0000 7f57e9e00700 1 -- 172.21.15.84:0/1906846250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57ec103580 msgr2=0x7f57ec199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:41:55.422 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.425+0000 7f57e9e00700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57ec103580 0x7f57ec199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:41:55.422 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.425+0000 7f57e9e00700 1 -- 172.21.15.84:0/1906846250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57ec1a1280 con 0x7f57ec102380 2024-04-28T12:41:55.422 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.425+0000 7f57e9400700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57ec103580 0x7f57ec199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:41:55.422 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.425+0000 7f57e9e00700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec102380 0x7f57ec199670 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00e970 tx=0x7f57dc00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:55.422 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.425+0000 7f57e2a00700 1 -- 172.21.15.84:0/1906846250 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc00cbd0 con 0x7f57ec102380 2024-04-28T12:41:55.422 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.425+0000 7f57e2a00700 1 -- 172.21.15.84:0/1906846250 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57dc0044d0 con 0x7f57ec102380 2024-04-28T12:41:55.422 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.425+0000 7f57f0a00700 1 -- 172.21.15.84:0/1906846250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57ec1a1570 con 0x7f57ec102380 2024-04-28T12:41:55.422 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.425+0000 7f57e2a00700 1 -- 172.21.15.84:0/1906846250 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc003d70 con 0x7f57ec102380 2024-04-28T12:41:55.422 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.425+0000 7f57f0a00700 1 -- 172.21.15.84:0/1906846250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57ec1a1ab0 con 0x7f57ec102380 2024-04-28T12:41:55.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.427+0000 7f57e2a00700 1 -- 172.21.15.84:0/1906846250 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f57dc021440 con 0x7f57ec102380 2024-04-28T12:41:55.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.427+0000 7f57f0a00700 1 -- 172.21.15.84:0/1906846250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57ec06a110 con 0x7f57ec102380 2024-04-28T12:41:55.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.427+0000 7f57e2a00700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f57d0039890 0x7f57d003bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:41:55.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.427+0000 7f57e2a00700 1 -- 172.21.15.84:0/1906846250 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f57dc04cc70 con 0x7f57ec102380 2024-04-28T12:41:55.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.428+0000 7f57e9400700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f57d0039890 0x7f57d003bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:41:55.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.428+0000 7f57e9400700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f57d0039890 0x7f57d003bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f57d8006010 tx=0x7f57d80037c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:41:55.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.433+0000 7f57e2a00700 1 -- 172.21.15.84:0/1906846250 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f57dc016bd0 con 0x7f57ec102380 2024-04-28T12:41:55.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:55.676+0000 7f57f0a00700 1 -- 172.21.15.84:0/1906846250 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f57ec06b880 con 0x7f57d0039890 2024-04-28T12:41:55.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:55.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:55.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.916 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:55 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:55.640+0000 7fafae308200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T12:41:55.916 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:55 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:55.729+0000 7fafae308200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T12:41:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.963 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.963 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.963 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.963 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.963 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.964 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:55.964 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:55.964 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:55.964 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:55 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:55.628+0000 7f1d79ccf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-28T12:41:55.964 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:55 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:55.795+0000 7f1d79ccf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-28T12:41:55.964 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:55 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:55.967+0000 7f1d79ccf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-28T12:41:56.293 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:55 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:55.965+0000 7fafae308200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T12:41:56.293 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:56 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:56.093+0000 7fafae308200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T12:41:56.293 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:56 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:56.185+0000 7fafae308200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T12:41:56.293 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:56 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:56.297+0000 7fafae308200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T12:41:56.544 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:56 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:56.252+0000 7f1d79ccf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-28T12:41:56.544 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:56 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:56.344+0000 7f1d79ccf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-28T12:41:56.727 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:56 smithi084 ceph-mon[28081]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:56.727 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:56 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:41:56.727 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:56 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:41:56.727 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:56 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:56.728 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:56 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:41:56.728 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:56 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:41:56.728 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:56 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:56.915 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:56 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:56.597+0000 7fafae308200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T12:41:56.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:56 smithi179 ceph-mon[63029]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:56.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:56 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:41:56.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:56 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:41:56.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:56 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:56.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:56 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:41:56.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:56 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:41:56.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:56 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:56.992 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:56 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:56.548+0000 7f1d79ccf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-28T12:41:56.993 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:56 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:56.740+0000 7f1d79ccf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-28T12:41:56.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:56 smithi161 ceph-mon[62387]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:56.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:56 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-28T12:41:56.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:56 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-28T12:41:56.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:56 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:56.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:56 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:41:56.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:56 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:41:56.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:56 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:57.556 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:57 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:57.181+0000 7fafae308200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T12:41:57.556 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 12:41:57 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z[64891]: 2024-04-28T12:41:57.391+0000 7fafae308200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T12:41:57.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:41:57.564+0000 7f57e2a00700 1 -- 172.21.15.84:0/1906846250 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f57dc01f340 con 0x7f57ec102380 2024-04-28T12:41:57.743 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:57 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:57.340+0000 7f1d79ccf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-28T12:41:57.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:57 smithi161 ceph-mon[62387]: Reconfiguring mgr.x (unknown last config time)... 2024-04-28T12:41:57.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:57 smithi161 ceph-mon[62387]: Reconfiguring daemon mgr.x on smithi084 2024-04-28T12:41:57.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:57 smithi161 ceph-mon[62387]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:57.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:57 smithi161 ceph-mon[62387]: from='mgr.? 172.21.15.179:0/2475564433' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-28T12:41:57.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:57 smithi161 ceph-mon[62387]: from='mgr.? 172.21.15.179:0/2475564433' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T12:41:57.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:57 smithi161 ceph-mon[62387]: Standby manager daemon z started 2024-04-28T12:41:57.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:57 smithi161 ceph-mon[62387]: from='mgr.? 172.21.15.179:0/2475564433' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-28T12:41:57.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:57 smithi161 ceph-mon[62387]: from='mgr.? 172.21.15.179:0/2475564433' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T12:41:57.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:57 smithi084 ceph-mon[28081]: Reconfiguring mgr.x (unknown last config time)... 2024-04-28T12:41:57.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:57 smithi084 ceph-mon[28081]: Reconfiguring daemon mgr.x on smithi084 2024-04-28T12:41:57.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:57 smithi084 ceph-mon[28081]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:57.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:57 smithi084 ceph-mon[28081]: from='mgr.? 172.21.15.179:0/2475564433' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-28T12:41:57.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:57 smithi084 ceph-mon[28081]: from='mgr.? 172.21.15.179:0/2475564433' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T12:41:57.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:57 smithi084 ceph-mon[28081]: Standby manager daemon z started 2024-04-28T12:41:57.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:57 smithi084 ceph-mon[28081]: from='mgr.? 172.21.15.179:0/2475564433' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-28T12:41:57.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:57 smithi084 ceph-mon[28081]: from='mgr.? 172.21.15.179:0/2475564433' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T12:41:57.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:57 smithi179 ceph-mon[63029]: Reconfiguring mgr.x (unknown last config time)... 2024-04-28T12:41:57.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:57 smithi179 ceph-mon[63029]: Reconfiguring daemon mgr.x on smithi084 2024-04-28T12:41:57.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:57 smithi179 ceph-mon[63029]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:41:57.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:57 smithi179 ceph-mon[63029]: from='mgr.? 172.21.15.179:0/2475564433' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-28T12:41:57.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:57 smithi179 ceph-mon[63029]: from='mgr.? 172.21.15.179:0/2475564433' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T12:41:57.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:57 smithi179 ceph-mon[63029]: Standby manager daemon z started 2024-04-28T12:41:57.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:57 smithi179 ceph-mon[63029]: from='mgr.? 172.21.15.179:0/2475564433' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-28T12:41:57.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:57 smithi179 ceph-mon[63029]: from='mgr.? 172.21.15.179:0/2475564433' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T12:41:58.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:57 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:57.761+0000 7f1d79ccf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-28T12:41:58.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:57 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:57.858+0000 7f1d79ccf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-28T12:41:58.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:58.060+0000 7f1d79ccf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-28T12:41:58.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:58 smithi179 ceph-mon[63029]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:58.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:58 smithi179 ceph-mon[63029]: mgrmap e14: x(active, since 88s), standbys: z 2024-04-28T12:41:58.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:58 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-28T12:41:58.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:58 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:58.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:58 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:58.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:58 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:58.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:58 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:58.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:58 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:58.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:41:58 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:58.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:58 smithi084 ceph-mon[28081]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:58.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:58 smithi084 ceph-mon[28081]: mgrmap e14: x(active, since 88s), standbys: z 2024-04-28T12:41:58.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:58 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-28T12:41:58.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:58 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:58.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:58 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:58.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:58 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:58.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:58 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:58.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:58 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:58.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:41:58 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:58.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:58.592+0000 7f1d79ccf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-28T12:41:58.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:58.683+0000 7f1d79ccf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-28T12:41:58.922 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-mon[62387]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:41:58.922 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-mon[62387]: mgrmap e14: x(active, since 88s), standbys: z 2024-04-28T12:41:58.923 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-28T12:41:58.923 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:58.923 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:58.923 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:41:58.923 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:41:58.923 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:41:58.923 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:41:59.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:58 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:58.926+0000 7f1d79ccf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-28T12:41:59.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:59 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:59.059+0000 7f1d79ccf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-28T12:41:59.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:59 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:59.154+0000 7f1d79ccf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-28T12:41:59.578 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:59 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:59.270+0000 7f1d79ccf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-28T12:41:59.993 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:41:59 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:41:59.582+0000 7f1d79ccf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-28T12:42:00.493 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:42:00 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:42:00.178+0000 7f1d79ccf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-28T12:42:00.493 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 12:42:00 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y[63765]: 2024-04-28T12:42:00.395+0000 7f1d79ccf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-28T12:42:00.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:00.578+0000 7f57e2a00700 1 -- 172.21.15.84:0/1906846250 <== mon.2 v2:172.21.15.161:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f57dc020ca0 con 0x7f57ec102380 2024-04-28T12:42:00.752 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:00 smithi161 ceph-mon[62387]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:00.752 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:00 smithi161 ceph-mon[62387]: from='mgr.? 172.21.15.161:0/3251735605' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-28T12:42:00.753 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:00 smithi161 ceph-mon[62387]: Standby manager daemon y started 2024-04-28T12:42:00.753 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:00 smithi161 ceph-mon[62387]: from='mgr.? 172.21.15.161:0/3251735605' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T12:42:00.753 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:00 smithi161 ceph-mon[62387]: from='mgr.? 172.21.15.161:0/3251735605' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-28T12:42:00.753 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:00 smithi161 ceph-mon[62387]: from='mgr.? 172.21.15.161:0/3251735605' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T12:42:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:00 smithi179 ceph-mon[63029]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:00 smithi179 ceph-mon[63029]: from='mgr.? 172.21.15.161:0/3251735605' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-28T12:42:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:00 smithi179 ceph-mon[63029]: Standby manager daemon y started 2024-04-28T12:42:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:00 smithi179 ceph-mon[63029]: from='mgr.? 172.21.15.161:0/3251735605' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T12:42:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:00 smithi179 ceph-mon[63029]: from='mgr.? 172.21.15.161:0/3251735605' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-28T12:42:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:00 smithi179 ceph-mon[63029]: from='mgr.? 172.21.15.161:0/3251735605' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T12:42:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:00 smithi084 ceph-mon[28081]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:00 smithi084 ceph-mon[28081]: from='mgr.? 172.21.15.161:0/3251735605' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-28T12:42:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:00 smithi084 ceph-mon[28081]: Standby manager daemon y started 2024-04-28T12:42:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:00 smithi084 ceph-mon[28081]: from='mgr.? 172.21.15.161:0/3251735605' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-28T12:42:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:00 smithi084 ceph-mon[28081]: from='mgr.? 172.21.15.161:0/3251735605' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-28T12:42:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:00 smithi084 ceph-mon[28081]: from='mgr.? 172.21.15.161:0/3251735605' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-28T12:42:01.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:01 smithi084 ceph-mon[28081]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-04-28T12:42:01.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:01 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-28T12:42:01.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:01 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1539542192' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2099232c-3140-41c4-83b4-89da6fa3a2fb"}]: dispatch 2024-04-28T12:42:01.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:01 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2099232c-3140-41c4-83b4-89da6fa3a2fb"}]: dispatch 2024-04-28T12:42:01.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:01 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2099232c-3140-41c4-83b4-89da6fa3a2fb"}]': finished 2024-04-28T12:42:01.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:01 smithi084 ceph-mon[28081]: osdmap e5: 1 total, 0 up, 1 in 2024-04-28T12:42:01.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:01 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:01.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:01 smithi179 ceph-mon[63029]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-04-28T12:42:01.953 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:01 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-28T12:42:01.953 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:01 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1539542192' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2099232c-3140-41c4-83b4-89da6fa3a2fb"}]: dispatch 2024-04-28T12:42:01.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:01 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2099232c-3140-41c4-83b4-89da6fa3a2fb"}]: dispatch 2024-04-28T12:42:01.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:01 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2099232c-3140-41c4-83b4-89da6fa3a2fb"}]': finished 2024-04-28T12:42:01.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:01 smithi179 ceph-mon[63029]: osdmap e5: 1 total, 0 up, 1 in 2024-04-28T12:42:01.954 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:01 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:01.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:01 smithi161 ceph-mon[62387]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-04-28T12:42:01.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:01 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-28T12:42:01.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:01 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1539542192' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2099232c-3140-41c4-83b4-89da6fa3a2fb"}]: dispatch 2024-04-28T12:42:01.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:01 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2099232c-3140-41c4-83b4-89da6fa3a2fb"}]: dispatch 2024-04-28T12:42:01.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:01 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2099232c-3140-41c4-83b4-89da6fa3a2fb"}]': finished 2024-04-28T12:42:01.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:01 smithi161 ceph-mon[62387]: osdmap e5: 1 total, 0 up, 1 in 2024-04-28T12:42:01.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:01 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:02.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:02 smithi179 ceph-mon[63029]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:02.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:02 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1819209582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:42:02.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:02 smithi084 ceph-mon[28081]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:02.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:02 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1819209582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:42:02.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:02 smithi161 ceph-mon[62387]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:02.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:02 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1819209582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:42:04.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:04 smithi179 ceph-mon[63029]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:04.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:04 smithi084 ceph-mon[28081]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:04.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:04 smithi161 ceph-mon[62387]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:06 smithi179 ceph-mon[63029]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:06 smithi084 ceph-mon[28081]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:06 smithi161 ceph-mon[62387]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:07.798 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:07 smithi084 ceph-mon[28081]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:07.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:07 smithi179 ceph-mon[63029]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:07.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:07 smithi161 ceph-mon[62387]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:10.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:10 smithi084 ceph-mon[28081]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:10.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-28T12:42:10.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:10.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:10 smithi179 ceph-mon[63029]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:10.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:10 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-28T12:42:10.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:10 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:10.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:10 smithi161 ceph-mon[62387]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:10.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:10 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-28T12:42:10.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:10 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:11.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:11 smithi084 ceph-mon[28081]: Deploying daemon osd.0 on smithi084 2024-04-28T12:42:11.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:11 smithi179 ceph-mon[63029]: Deploying daemon osd.0 on smithi084 2024-04-28T12:42:11.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:11 smithi161 ceph-mon[62387]: Deploying daemon osd.0 on smithi084 2024-04-28T12:42:12.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:12 smithi084 ceph-mon[28081]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:12.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:12 smithi179 ceph-mon[63029]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:12.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:12 smithi161 ceph-mon[62387]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:14.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:14 smithi084 ceph-mon[28081]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:14 smithi179 ceph-mon[63029]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:14.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:14 smithi161 ceph-mon[62387]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:16.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:16 smithi084 ceph-mon[28081]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:16.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:16 smithi179 ceph-mon[63029]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:16.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:16 smithi161 ceph-mon[62387]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:18.666 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:18 smithi084 ceph-mon[28081]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:18.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:18.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:18.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:18.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:18 smithi179 ceph-mon[63029]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:18.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:18.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:18.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:18.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:18 smithi161 ceph-mon[62387]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:18.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:18.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:18.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:20 smithi179 ceph-mon[63029]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:20 smithi084 ceph-mon[28081]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:20.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:20 smithi161 ceph-mon[62387]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:22.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:22 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:22.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:22 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:22.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:22 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:22.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:22 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:22.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:22 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:22.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:22 smithi084 ceph-mon[28081]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:22.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:22 smithi084 ceph-mon[28081]: from='osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-28T12:42:22.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:22 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:22.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:22 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:22.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:22 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:22.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:22 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:22.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:22 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:22.666 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:22 smithi179 ceph-mon[63029]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:22.666 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:22 smithi179 ceph-mon[63029]: from='osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-28T12:42:22.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.731+0000 7f57e2a00700 1 -- 172.21.15.84:0/1906846250 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f57ec06b880 con 0x7f57d0039890 2024-04-28T12:42:22.729 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 0 on host 'smithi084' 2024-04-28T12:42:22.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.734+0000 7f57f0a00700 1 -- 172.21.15.84:0/1906846250 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f57d0039890 msgr2=0x7f57d003bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:22.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.734+0000 7f57f0a00700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f57d0039890 0x7f57d003bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f57d8006010 tx=0x7f57d80037c0 comp rx=0 tx=0).stop 2024-04-28T12:42:22.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.734+0000 7f57f0a00700 1 -- 172.21.15.84:0/1906846250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec102380 msgr2=0x7f57ec199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:22.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.734+0000 7f57f0a00700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec102380 0x7f57ec199670 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00e970 tx=0x7f57dc00ed30 comp rx=0 tx=0).stop 2024-04-28T12:42:22.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.735+0000 7f57f0a00700 1 -- 172.21.15.84:0/1906846250 shutdown_connections 2024-04-28T12:42:22.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.735+0000 7f57f0a00700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f57d0039890 0x7f57d003bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:22.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.735+0000 7f57f0a00700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec102380 0x7f57ec199670 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:22.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.735+0000 7f57f0a00700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57ec103f20 0x7f57ec1a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:22.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.735+0000 7f57f0a00700 1 --2- 172.21.15.84:0/1906846250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57ec103580 0x7f57ec199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:22.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.735+0000 7f57f0a00700 1 -- 172.21.15.84:0/1906846250 >> 172.21.15.84:0/1906846250 conn(0x7f57ec0fd920 msgr2=0x7f57ec06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:22.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.735+0000 7f57f0a00700 1 -- 172.21.15.84:0/1906846250 shutdown_connections 2024-04-28T12:42:22.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:42:22.735+0000 7f57f0a00700 1 -- 172.21.15.84:0/1906846250 wait complete. 2024-04-28T12:42:22.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:22 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:22.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:22 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:22.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:22 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:22.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:22 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:22.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:22 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:22.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:22 smithi161 ceph-mon[62387]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:22.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:22 smithi161 ceph-mon[62387]: from='osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-28T12:42:23.415 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:23 smithi084 ceph-mon[28081]: from='osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-28T12:42:23.415 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:23 smithi084 ceph-mon[28081]: osdmap e6: 1 total, 0 up, 1 in 2024-04-28T12:42:23.415 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:23 smithi084 ceph-mon[28081]: from='osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-04-28T12:42:23.415 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:23 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:23.415 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:23 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:23.415 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:23 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:23.415 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:23 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:23.494 DEBUG:teuthology.orchestra.run.smithi084:osd.0> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.0.service 2024-04-28T12:42:23.499 INFO:tasks.cephadm:Deploying osd.1 on smithi161 with /dev/vg_nvme/lv_4... 2024-04-28T12:42:23.499 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-28T12:42:23.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:23 smithi179 ceph-mon[63029]: from='osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-28T12:42:23.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:23 smithi179 ceph-mon[63029]: osdmap e6: 1 total, 0 up, 1 in 2024-04-28T12:42:23.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:23 smithi179 ceph-mon[63029]: from='osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-04-28T12:42:23.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:23 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:23.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:23 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:23.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:23 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:23.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:23 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:23.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:23 smithi161 ceph-mon[62387]: from='osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-28T12:42:23.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:23 smithi161 ceph-mon[62387]: osdmap e6: 1 total, 0 up, 1 in 2024-04-28T12:42:23.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:23 smithi161 ceph-mon[62387]: from='osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-04-28T12:42:23.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:23 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:23.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:23 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:23.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:23 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:23.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:23 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:42:24.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:24 smithi179 ceph-mon[63029]: from='osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-04-28T12:42:24.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:24 smithi179 ceph-mon[63029]: osdmap e7: 1 total, 0 up, 1 in 2024-04-28T12:42:24.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:24 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:24.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:24 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:24.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:24 smithi179 ceph-mon[63029]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:24.472 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:24 smithi161 ceph-mon[62387]: from='osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-04-28T12:42:24.472 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:24 smithi161 ceph-mon[62387]: osdmap e7: 1 total, 0 up, 1 in 2024-04-28T12:42:24.472 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:24 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:24.472 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:24 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:24.473 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:24 smithi161 ceph-mon[62387]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:24.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:24 smithi084 ceph-mon[28081]: from='osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-04-28T12:42:24.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:24 smithi084 ceph-mon[28081]: osdmap e7: 1 total, 0 up, 1 in 2024-04-28T12:42:24.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:24 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:24.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:24 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:24.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:24 smithi084 ceph-mon[28081]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-28T12:42:25.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:25 smithi161 ceph-mon[62387]: purged_snaps scrub starts 2024-04-28T12:42:25.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:25 smithi161 ceph-mon[62387]: purged_snaps scrub ok 2024-04-28T12:42:25.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:25 smithi161 ceph-mon[62387]: osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287] boot 2024-04-28T12:42:25.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:25 smithi161 ceph-mon[62387]: osdmap e8: 1 total, 1 up, 1 in 2024-04-28T12:42:25.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:25 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:25.503 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-28T12:42:25.503 INFO:teuthology.orchestra.run.smithi161.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-28T12:42:25.503 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10+0 records in 2024-04-28T12:42:25.504 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2024-04-28T12:42:25.504 INFO:teuthology.orchestra.run.smithi161.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0118371 s, 886 MB/s 2024-04-28T12:42:25.504 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping successful for: 2024-04-28T12:42:25.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:25 smithi084 ceph-mon[28081]: purged_snaps scrub starts 2024-04-28T12:42:25.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:25 smithi084 ceph-mon[28081]: purged_snaps scrub ok 2024-04-28T12:42:25.595 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:25 smithi084 ceph-mon[28081]: osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287] boot 2024-04-28T12:42:25.595 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:25 smithi084 ceph-mon[28081]: osdmap e8: 1 total, 1 up, 1 in 2024-04-28T12:42:25.595 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:25 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:25.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:25 smithi179 ceph-mon[63029]: purged_snaps scrub starts 2024-04-28T12:42:25.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:25 smithi179 ceph-mon[63029]: purged_snaps scrub ok 2024-04-28T12:42:25.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:25 smithi179 ceph-mon[63029]: osd.0 [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287] boot 2024-04-28T12:42:25.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:25 smithi179 ceph-mon[63029]: osdmap e8: 1 total, 1 up, 1 in 2024-04-28T12:42:25.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-28T12:42:26.114 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi161:vg_nvme/lv_4 2024-04-28T12:42:26.356 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:42:26.392 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:26 smithi161 ceph-mon[62387]: osdmap e9: 1 total, 1 up, 1 in 2024-04-28T12:42:26.392 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:26 smithi161 ceph-mon[62387]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:26.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:26 smithi179 ceph-mon[63029]: osdmap e9: 1 total, 1 up, 1 in 2024-04-28T12:42:26.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:26 smithi179 ceph-mon[63029]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:26.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:26 smithi084 ceph-mon[28081]: osdmap e9: 1 total, 1 up, 1 in 2024-04-28T12:42:26.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:26 smithi084 ceph-mon[28081]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:27.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.844+0000 7fa1d524a700 1 -- 172.21.15.161:0/2853295108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1d0103480 msgr2=0x7fa1d01038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:27.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.844+0000 7fa1d524a700 1 --2- 172.21.15.161:0/2853295108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1d0103480 0x7fa1d01038a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa1c40099c0 tx=0x7fa1c4009cd0 comp rx=0 tx=0).stop 2024-04-28T12:42:27.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.844+0000 7fa1d524a700 1 -- 172.21.15.161:0/2853295108 shutdown_connections 2024-04-28T12:42:27.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.844+0000 7fa1d524a700 1 --2- 172.21.15.161:0/2853295108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1d0103480 0x7fa1d01038a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:27.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.844+0000 7fa1d524a700 1 --2- 172.21.15.161:0/2853295108 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa1d0105020 0x7fa1d010b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:27.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.844+0000 7fa1d524a700 1 --2- 172.21.15.161:0/2853295108 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1d0104680 0x7fa1d0104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:27.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.844+0000 7fa1d524a700 1 -- 172.21.15.161:0/2853295108 >> 172.21.15.161:0/2853295108 conn(0x7fa1d00fea20 msgr2=0x7fa1d0100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:27.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.844+0000 7fa1d524a700 1 -- 172.21.15.161:0/2853295108 shutdown_connections 2024-04-28T12:42:27.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.844+0000 7fa1d524a700 1 -- 172.21.15.161:0/2853295108 wait complete. 2024-04-28T12:42:27.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.845+0000 7fa1d524a700 1 Processor -- start 2024-04-28T12:42:27.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.845+0000 7fa1d524a700 1 -- start start 2024-04-28T12:42:27.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1d524a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1d0103480 0x7fa1d0199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:27.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1d524a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa1d0104680 0x7fa1d0199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:27.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1d524a700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1d0105020 0x7fa1d01a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:27.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1d524a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d010efa0 con 0x7fa1d0104680 2024-04-28T12:42:27.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1d524a700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d010ee20 con 0x7fa1d0103480 2024-04-28T12:42:27.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1d524a700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d010f120 con 0x7fa1d0105020 2024-04-28T12:42:27.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1ced9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1d0103480 0x7fa1d0199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1ced9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1d0103480 0x7fa1d0199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41548/0 (socket says 172.21.15.161:41548) 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1ced9d700 1 -- 172.21.15.161:0/4003382396 learned_addr learned my addr 172.21.15.161:0/4003382396 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1ced9d700 1 -- 172.21.15.161:0/4003382396 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1d0105020 msgr2=0x7fa1d01a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1ced9d700 1 --2- 172.21.15.161:0/4003382396 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1d0105020 0x7fa1d01a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1ced9d700 1 -- 172.21.15.161:0/4003382396 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa1d0104680 msgr2=0x7fa1d0199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1ced9d700 1 --2- 172.21.15.161:0/4003382396 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa1d0104680 0x7fa1d0199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1ced9d700 1 -- 172.21.15.161:0/4003382396 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d01a12d0 con 0x7fa1d0103480 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1ced9d700 1 --2- 172.21.15.161:0/4003382396 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1d0103480 0x7fa1d0199670 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400fa00 tx=0x7fa1c400fae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1b7fff700 1 -- 172.21.15.161:0/4003382396 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4027070 con 0x7fa1d0103480 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1d524a700 1 -- 172.21.15.161:0/4003382396 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d01a1560 con 0x7fa1d0103480 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1b7fff700 1 -- 172.21.15.161:0/4003382396 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1c4005940 con 0x7fa1d0103480 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1b7fff700 1 -- 172.21.15.161:0/4003382396 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4017410 con 0x7fa1d0103480 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.846+0000 7fa1d524a700 1 -- 172.21.15.161:0/4003382396 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d01a1ac0 con 0x7fa1d0103480 2024-04-28T12:42:27.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.847+0000 7fa1d524a700 1 -- 172.21.15.161:0/4003382396 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1d006a110 con 0x7fa1d0103480 2024-04-28T12:42:27.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.848+0000 7fa1b7fff700 1 -- 172.21.15.161:0/4003382396 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fa1c402a070 con 0x7fa1d0103480 2024-04-28T12:42:27.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.849+0000 7fa1b7fff700 1 --2- 172.21.15.161:0/4003382396 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa1b80a3bb0 0x7fa1b80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:27.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.849+0000 7fa1b7fff700 1 -- 172.21.15.161:0/4003382396 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fa1c40cf620 con 0x7fa1d0103480 2024-04-28T12:42:27.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.849+0000 7fa1ce59c700 1 --2- 172.21.15.161:0/4003382396 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa1b80a3bb0 0x7fa1b80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:27.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.850+0000 7fa1ce59c700 1 --2- 172.21.15.161:0/4003382396 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa1b80a3bb0 0x7fa1b80a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005950 tx=0x7fa1c00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:27.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:27.854+0000 7fa1b7fff700 1 -- 172.21.15.161:0/4003382396 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa1c4013070 con 0x7fa1d0103480 2024-04-28T12:42:28.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:28.059+0000 7fa1d524a700 1 -- 172.21.15.161:0/4003382396 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa1d0063860 con 0x7fa1b80a3bb0 2024-04-28T12:42:28.819 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:28 smithi161 ceph-mon[62387]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:28.819 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:42:28.819 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:42:28.819 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:28.819 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:28.819 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:28.819 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:42:28.819 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:28.820 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:28.820 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:28.820 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:28.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:28 smithi179 ceph-mon[63029]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:28.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:42:28.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:42:28.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:28.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:28.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:28.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:42:28.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:28.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:28.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:28.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:28 smithi084 ceph-mon[28081]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:42:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:42:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:42:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:28.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:29.802 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:29 smithi161 ceph-mon[62387]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:29.802 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:29 smithi161 ceph-mon[62387]: Detected new or changed devices on smithi084 2024-04-28T12:42:29.802 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:29 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi084 to 17089M 2024-04-28T12:42:29.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:29 smithi179 ceph-mon[63029]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:29.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:29 smithi179 ceph-mon[63029]: Detected new or changed devices on smithi084 2024-04-28T12:42:29.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:29 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi084 to 17089M 2024-04-28T12:42:29.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:29 smithi084 ceph-mon[28081]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:29.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:29 smithi084 ceph-mon[28081]: Detected new or changed devices on smithi084 2024-04-28T12:42:29.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:29 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi084 to 17089M 2024-04-28T12:42:30.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:30 smithi084 ceph-mon[28081]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:30.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:30 smithi179 ceph-mon[63029]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:30.961 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:30 smithi161 ceph-mon[62387]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:31.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:31 smithi179 ceph-mon[63029]: from='client.? 172.21.15.161:0/3482732892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee0ea48d-5933-4d73-a6ee-a9028badef12"}]: dispatch 2024-04-28T12:42:31.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:31 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee0ea48d-5933-4d73-a6ee-a9028badef12"}]: dispatch 2024-04-28T12:42:31.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:31 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee0ea48d-5933-4d73-a6ee-a9028badef12"}]': finished 2024-04-28T12:42:31.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:31 smithi179 ceph-mon[63029]: osdmap e10: 2 total, 1 up, 2 in 2024-04-28T12:42:31.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:31 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:31.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:31 smithi179 ceph-mon[63029]: from='client.? 172.21.15.161:0/4270776518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:42:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:31 smithi084 ceph-mon[28081]: from='client.? 172.21.15.161:0/3482732892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee0ea48d-5933-4d73-a6ee-a9028badef12"}]: dispatch 2024-04-28T12:42:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:31 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee0ea48d-5933-4d73-a6ee-a9028badef12"}]: dispatch 2024-04-28T12:42:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:31 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee0ea48d-5933-4d73-a6ee-a9028badef12"}]': finished 2024-04-28T12:42:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:31 smithi084 ceph-mon[28081]: osdmap e10: 2 total, 1 up, 2 in 2024-04-28T12:42:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:31 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:31 smithi084 ceph-mon[28081]: from='client.? 172.21.15.161:0/4270776518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:42:31.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:31 smithi161 ceph-mon[62387]: from='client.? 172.21.15.161:0/3482732892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee0ea48d-5933-4d73-a6ee-a9028badef12"}]: dispatch 2024-04-28T12:42:31.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:31 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee0ea48d-5933-4d73-a6ee-a9028badef12"}]: dispatch 2024-04-28T12:42:31.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:31 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee0ea48d-5933-4d73-a6ee-a9028badef12"}]': finished 2024-04-28T12:42:31.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:31 smithi161 ceph-mon[62387]: osdmap e10: 2 total, 1 up, 2 in 2024-04-28T12:42:31.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:31 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:31.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:31 smithi161 ceph-mon[62387]: from='client.? 172.21.15.161:0/4270776518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:42:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:32 smithi179 ceph-mon[63029]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:32.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:32 smithi084 ceph-mon[28081]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:32.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:32 smithi161 ceph-mon[62387]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:34.842 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:34 smithi161 ceph-mon[62387]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:34 smithi179 ceph-mon[63029]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:34.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:34 smithi084 ceph-mon[28081]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:36.682 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:36 smithi161 ceph-mon[62387]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:36 smithi179 ceph-mon[63029]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:36.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:36 smithi084 ceph-mon[28081]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:38 smithi179 ceph-mon[63029]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-28T12:42:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:38 smithi084 ceph-mon[28081]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-28T12:42:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:38.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:38 smithi161 ceph-mon[62387]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:38.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-28T12:42:38.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:39.688 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:39 smithi161 ceph-mon[62387]: Deploying daemon osd.1 on smithi161 2024-04-28T12:42:39.853 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:39 smithi084 ceph-mon[28081]: Deploying daemon osd.1 on smithi161 2024-04-28T12:42:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:39 smithi179 ceph-mon[63029]: Deploying daemon osd.1 on smithi161 2024-04-28T12:42:40.616 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:40 smithi161 ceph-mon[62387]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:40 smithi179 ceph-mon[63029]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:40.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:40 smithi084 ceph-mon[28081]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:42.641 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:42 smithi161 ceph-mon[62387]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:42.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:42 smithi179 ceph-mon[63029]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:42.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:42 smithi084 ceph-mon[28081]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:44.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:44.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:44.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:44.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:44 smithi161 ceph-mon[62387]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:44.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:44.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:44.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:44.666 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:44 smithi179 ceph-mon[63029]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:44.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:44.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:44.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:44.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:44 smithi084 ceph-mon[28081]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:46.809 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:46 smithi161 ceph-mon[62387]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:46.809 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:46.809 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:46.809 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:46.810 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:46.810 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:46.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.896+0000 7fa1b7fff700 1 -- 172.21.15.161:0/4003382396 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa1d0063860 con 0x7fa1b80a3bb0 2024-04-28T12:42:46.892 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 1 on host 'smithi161' 2024-04-28T12:42:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.899+0000 7fa1d524a700 1 -- 172.21.15.161:0/4003382396 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa1b80a3bb0 msgr2=0x7fa1b80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.899+0000 7fa1d524a700 1 --2- 172.21.15.161:0/4003382396 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa1b80a3bb0 0x7fa1b80a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005950 tx=0x7fa1c00058e0 comp rx=0 tx=0).stop 2024-04-28T12:42:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.899+0000 7fa1d524a700 1 -- 172.21.15.161:0/4003382396 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1d0103480 msgr2=0x7fa1d0199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.899+0000 7fa1d524a700 1 --2- 172.21.15.161:0/4003382396 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1d0103480 0x7fa1d0199670 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400fa00 tx=0x7fa1c400fae0 comp rx=0 tx=0).stop 2024-04-28T12:42:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.899+0000 7fa1d524a700 1 -- 172.21.15.161:0/4003382396 shutdown_connections 2024-04-28T12:42:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.899+0000 7fa1d524a700 1 --2- 172.21.15.161:0/4003382396 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa1b80a3bb0 0x7fa1b80a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.899+0000 7fa1d524a700 1 --2- 172.21.15.161:0/4003382396 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1d0103480 0x7fa1d0199670 secure :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400fa00 tx=0x7fa1c400fae0 comp rx=0 tx=0).stop 2024-04-28T12:42:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.899+0000 7fa1d524a700 1 --2- 172.21.15.161:0/4003382396 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa1d0105020 0x7fa1d01a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.899+0000 7fa1d524a700 1 --2- 172.21.15.161:0/4003382396 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa1d0104680 0x7fa1d0199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.899+0000 7fa1d524a700 1 -- 172.21.15.161:0/4003382396 >> 172.21.15.161:0/4003382396 conn(0x7fa1d00fea20 msgr2=0x7fa1d0107ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.899+0000 7fa1d524a700 1 -- 172.21.15.161:0/4003382396 shutdown_connections 2024-04-28T12:42:46.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:42:46.899+0000 7fa1d524a700 1 -- 172.21.15.161:0/4003382396 wait complete. 2024-04-28T12:42:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:46 smithi179 ceph-mon[63029]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:46 smithi084 ceph-mon[28081]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:47.496 DEBUG:teuthology.orchestra.run.smithi161:osd.1> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.1.service 2024-04-28T12:42:47.499 INFO:tasks.cephadm:Deploying osd.2 on smithi179 with /dev/vg_nvme/lv_4... 2024-04-28T12:42:47.499 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-28T12:42:47.743 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:42:48.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:48.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:48.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:48.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:47 smithi179 ceph-mon[63029]: from='osd.1 [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T12:42:48.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:47 smithi179 ceph-mon[63029]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T12:42:48.149 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:47 smithi179 ceph-mon[63029]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:47 smithi084 ceph-mon[28081]: from='osd.1 [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T12:42:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:47 smithi084 ceph-mon[28081]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T12:42:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:47 smithi084 ceph-mon[28081]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:48.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:48.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:48.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:42:48.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:47 smithi161 ceph-mon[62387]: from='osd.1 [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T12:42:48.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:47 smithi161 ceph-mon[62387]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-28T12:42:48.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:47 smithi161 ceph-mon[62387]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:48.911 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:48 smithi161 ceph-mon[62387]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-28T12:42:48.911 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:48 smithi161 ceph-mon[62387]: osdmap e11: 2 total, 1 up, 2 in 2024-04-28T12:42:48.911 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:48 smithi161 ceph-mon[62387]: from='osd.1 [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:42:48.911 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:48 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:48.911 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:48 smithi161 ceph-mon[62387]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:42:49.039 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:48 smithi179 ceph-mon[63029]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-28T12:42:49.039 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:48 smithi179 ceph-mon[63029]: osdmap e11: 2 total, 1 up, 2 in 2024-04-28T12:42:49.039 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:48 smithi179 ceph-mon[63029]: from='osd.1 [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:42:49.039 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:48 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:49.039 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:48 smithi179 ceph-mon[63029]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:42:49.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:48 smithi084 ceph-mon[28081]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-28T12:42:49.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:48 smithi084 ceph-mon[28081]: osdmap e11: 2 total, 1 up, 2 in 2024-04-28T12:42:49.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:48 smithi084 ceph-mon[28081]: from='osd.1 [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:42:49.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:48 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:49.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:48 smithi084 ceph-mon[28081]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:42:49.611 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-28T12:42:49.611 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-28T12:42:49.611 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-04-28T12:42:49.611 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-04-28T12:42:49.612 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115092 s, 911 MB/s 2024-04-28T12:42:49.612 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-04-28T12:42:49.698 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 12:42:49 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1[66958]: 2024-04-28T12:42:49.330+0000 7f6c82853700 -1 osd.1 0 waiting for initial osdmap 2024-04-28T12:42:49.698 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 12:42:49 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1[66958]: 2024-04-28T12:42:49.333+0000 7f6c79e3d700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:42:49.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:49 smithi161 ceph-mon[62387]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-04-28T12:42:49.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:49 smithi161 ceph-mon[62387]: osdmap e12: 2 total, 1 up, 2 in 2024-04-28T12:42:49.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:49.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:49.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:49 smithi161 ceph-mon[62387]: from='osd.1 ' entity='osd.1' 2024-04-28T12:42:49.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:49 smithi161 ceph-mon[62387]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:50.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:49 smithi179 ceph-mon[63029]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-04-28T12:42:50.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:49 smithi179 ceph-mon[63029]: osdmap e12: 2 total, 1 up, 2 in 2024-04-28T12:42:50.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:50.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:50.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:49 smithi179 ceph-mon[63029]: from='osd.1 ' entity='osd.1' 2024-04-28T12:42:50.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:49 smithi179 ceph-mon[63029]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:50.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:49 smithi084 ceph-mon[28081]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-04-28T12:42:50.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:49 smithi084 ceph-mon[28081]: osdmap e12: 2 total, 1 up, 2 in 2024-04-28T12:42:50.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:50.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:50.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:49 smithi084 ceph-mon[28081]: from='osd.1 ' entity='osd.1' 2024-04-28T12:42:50.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:49 smithi084 ceph-mon[28081]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-28T12:42:50.302 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi179:vg_nvme/lv_4 2024-04-28T12:42:50.545 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:42:51.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: purged_snaps scrub starts 2024-04-28T12:42:51.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: purged_snaps scrub ok 2024-04-28T12:42:51.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:51.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: osd.1 [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633] boot 2024-04-28T12:42:51.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: osdmap e13: 2 total, 2 up, 2 in 2024-04-28T12:42:51.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:51.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: Detected new or changed devices on smithi161 2024-04-28T12:42:51.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:51.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:51.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:42:51.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi161 to 17088M 2024-04-28T12:42:51.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:51.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:51.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:51.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:51.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: purged_snaps scrub starts 2024-04-28T12:42:51.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: purged_snaps scrub ok 2024-04-28T12:42:51.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:51.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: osd.1 [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633] boot 2024-04-28T12:42:51.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: osdmap e13: 2 total, 2 up, 2 in 2024-04-28T12:42:51.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:51.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: Detected new or changed devices on smithi161 2024-04-28T12:42:51.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:51.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:51.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:42:51.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi161 to 17088M 2024-04-28T12:42:51.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:51.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:51.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:51.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:51.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: purged_snaps scrub starts 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: purged_snaps scrub ok 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: osd.1 [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633] boot 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: osdmap e13: 2 total, 2 up, 2 in 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: Detected new or changed devices on smithi161 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi161 to 17088M 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:42:51.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:42:52.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.031+0000 7fab1b375700 1 -- 172.21.15.179:0/298759956 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14103f20 msgr2=0x7fab1410a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:52.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.031+0000 7fab1b375700 1 --2- 172.21.15.179:0/298759956 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14103f20 0x7fab1410a7d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00b0a0 tx=0x7fab0c00b3b0 comp rx=0 tx=0).stop 2024-04-28T12:42:52.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.031+0000 7fab1b375700 1 -- 172.21.15.179:0/298759956 shutdown_connections 2024-04-28T12:42:52.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.031+0000 7fab1b375700 1 --2- 172.21.15.179:0/298759956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab14102380 0x7fab141027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:52.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.031+0000 7fab1b375700 1 --2- 172.21.15.179:0/298759956 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14103f20 0x7fab1410a7d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:52.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.031+0000 7fab1b375700 1 --2- 172.21.15.179:0/298759956 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab14103580 0x7fab141039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:52.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.031+0000 7fab1b375700 1 -- 172.21.15.179:0/298759956 >> 172.21.15.179:0/298759956 conn(0x7fab140fd920 msgr2=0x7fab140ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:42:52.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab1b375700 1 -- 172.21.15.179:0/298759956 shutdown_connections 2024-04-28T12:42:52.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab1b375700 1 -- 172.21.15.179:0/298759956 wait complete. 2024-04-28T12:42:52.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab1b375700 1 Processor -- start 2024-04-28T12:42:52.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab1b375700 1 -- start start 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab1b375700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab14102380 0x7fab1406c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab1b375700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14103580 0x7fab1419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab1b375700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab14103f20 0x7fab141a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab1b375700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fab1410df60 con 0x7fab14103f20 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab1b375700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fab1410dde0 con 0x7fab14102380 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab1b375700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fab1410e0e0 con 0x7fab14103580 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab18910700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14103580 0x7fab1419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab18910700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14103580 0x7fab1419e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:34820/0 (socket says 172.21.15.179:34820) 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.032+0000 7fab18910700 1 -- 172.21.15.179:0/2794183982 learned_addr learned my addr 172.21.15.179:0/2794183982 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.033+0000 7fab19111700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab14102380 0x7fab1406c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.033+0000 7fab18910700 1 -- 172.21.15.179:0/2794183982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab14102380 msgr2=0x7fab1406c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.033+0000 7fab18910700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab14102380 0x7fab1406c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:52.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.033+0000 7fab18910700 1 -- 172.21.15.179:0/2794183982 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab14103f20 msgr2=0x7fab141a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:42:52.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.033+0000 7fab18910700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab14103f20 0x7fab141a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:42:52.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.033+0000 7fab18910700 1 -- 172.21.15.179:0/2794183982 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab141a1360 con 0x7fab14103580 2024-04-28T12:42:52.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.033+0000 7fab18910700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14103580 0x7fab1419e400 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fab0800b530 tx=0x7fab0800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:52.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.033+0000 7fab067fc700 1 -- 172.21.15.179:0/2794183982 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08015020 con 0x7fab14103580 2024-04-28T12:42:52.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.033+0000 7fab067fc700 1 -- 172.21.15.179:0/2794183982 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab08011ce0 con 0x7fab14103580 2024-04-28T12:42:52.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.033+0000 7fab1b375700 1 -- 172.21.15.179:0/2794183982 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab14198230 con 0x7fab14103580 2024-04-28T12:42:52.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.033+0000 7fab067fc700 1 -- 172.21.15.179:0/2794183982 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08010d20 con 0x7fab14103580 2024-04-28T12:42:52.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.033+0000 7fab1b375700 1 -- 172.21.15.179:0/2794183982 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab141987f0 con 0x7fab14103580 2024-04-28T12:42:52.034 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.034+0000 7fab1b375700 1 -- 172.21.15.179:0/2794183982 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab1406a110 con 0x7fab14103580 2024-04-28T12:42:52.034 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.035+0000 7fab067fc700 1 -- 172.21.15.179:0/2794183982 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fab08018020 con 0x7fab14103580 2024-04-28T12:42:52.034 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.036+0000 7fab067fc700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fab000a3bb0 0x7fab000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:42:52.034 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.036+0000 7fab067fc700 1 -- 172.21.15.179:0/2794183982 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fab080ce3b0 con 0x7fab14103580 2024-04-28T12:42:52.034 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:51 smithi179 ceph-mon[63029]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:42:52.034 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.036+0000 7fab19111700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fab000a3bb0 0x7fab000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:42:52.035 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.037+0000 7fab19111700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fab000a3bb0 0x7fab000a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fab10006fd0 tx=0x7fab10008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:42:52.037 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.040+0000 7fab067fc700 1 -- 172.21.15.179:0/2794183982 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fab0801f020 con 0x7fab14103580 2024-04-28T12:42:52.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:51 smithi084 ceph-mon[28081]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:42:52.240 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:42:52.243+0000 7fab1b375700 1 -- 172.21.15.179:0/2794183982 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fab14109260 con 0x7fab000a3bb0 2024-04-28T12:42:52.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:51 smithi161 ceph-mon[62387]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:42:53.162 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:52 smithi179 ceph-mon[63029]: osdmap e14: 2 total, 2 up, 2 in 2024-04-28T12:42:53.162 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:52 smithi179 ceph-mon[63029]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:53.162 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:42:53.162 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:42:53.162 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:53.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:52 smithi084 ceph-mon[28081]: osdmap e14: 2 total, 2 up, 2 in 2024-04-28T12:42:53.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:52 smithi084 ceph-mon[28081]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:53.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:42:53.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:42:53.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:53.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:52 smithi161 ceph-mon[62387]: osdmap e14: 2 total, 2 up, 2 in 2024-04-28T12:42:53.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:52 smithi161 ceph-mon[62387]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:42:53.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:42:53.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:42:53.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:42:54.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:53 smithi179 ceph-mon[63029]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:42:54.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:53 smithi084 ceph-mon[28081]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:42:54.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:53 smithi161 ceph-mon[62387]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:42:55.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:54 smithi179 ceph-mon[63029]: from='client.? 172.21.15.179:0/2325959631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac1f166c-e9a5-483c-9551-026ed03d1564"}]: dispatch 2024-04-28T12:42:55.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:54 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac1f166c-e9a5-483c-9551-026ed03d1564"}]: dispatch 2024-04-28T12:42:55.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:54 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac1f166c-e9a5-483c-9551-026ed03d1564"}]': finished 2024-04-28T12:42:55.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:54 smithi179 ceph-mon[63029]: osdmap e15: 3 total, 2 up, 3 in 2024-04-28T12:42:55.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:54 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:42:55.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:54 smithi084 ceph-mon[28081]: from='client.? 172.21.15.179:0/2325959631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac1f166c-e9a5-483c-9551-026ed03d1564"}]: dispatch 2024-04-28T12:42:55.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:54 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac1f166c-e9a5-483c-9551-026ed03d1564"}]: dispatch 2024-04-28T12:42:55.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:54 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac1f166c-e9a5-483c-9551-026ed03d1564"}]': finished 2024-04-28T12:42:55.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:54 smithi084 ceph-mon[28081]: osdmap e15: 3 total, 2 up, 3 in 2024-04-28T12:42:55.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:54 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:42:55.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:54 smithi161 ceph-mon[62387]: from='client.? 172.21.15.179:0/2325959631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac1f166c-e9a5-483c-9551-026ed03d1564"}]: dispatch 2024-04-28T12:42:55.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:54 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac1f166c-e9a5-483c-9551-026ed03d1564"}]: dispatch 2024-04-28T12:42:55.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:54 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac1f166c-e9a5-483c-9551-026ed03d1564"}]': finished 2024-04-28T12:42:55.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:54 smithi161 ceph-mon[62387]: osdmap e15: 3 total, 2 up, 3 in 2024-04-28T12:42:55.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:54 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:42:56.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:55 smithi161 ceph-mon[62387]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:42:56.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:55 smithi161 ceph-mon[62387]: from='client.? 172.21.15.179:0/541757284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:42:56.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:55 smithi179 ceph-mon[63029]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:42:56.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:55 smithi179 ceph-mon[63029]: from='client.? 172.21.15.179:0/541757284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:42:56.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:55 smithi084 ceph-mon[28081]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:42:56.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:55 smithi084 ceph-mon[28081]: from='client.? 172.21.15.179:0/541757284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:42:58.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:42:58 smithi179 ceph-mon[63029]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:42:58.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:42:58 smithi084 ceph-mon[28081]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:42:58.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:42:58 smithi161 ceph-mon[62387]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:00.696 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:00 smithi179 ceph-mon[63029]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:00.818 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:00 smithi161 ceph-mon[62387]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:00 smithi084 ceph-mon[28081]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:02.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:02 smithi179 ceph-mon[63029]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:02.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:02 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-28T12:43:02.881 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:02 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:02.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:02 smithi084 ceph-mon[28081]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:02.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:02 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-28T12:43:02.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:02 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:02.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:02 smithi161 ceph-mon[62387]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:02.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:02 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-28T12:43:02.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:02 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:03.818 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:03 smithi179 ceph-mon[63029]: Deploying daemon osd.2 on smithi179 2024-04-28T12:43:03.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:03 smithi084 ceph-mon[28081]: Deploying daemon osd.2 on smithi179 2024-04-28T12:43:03.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:03 smithi161 ceph-mon[62387]: Deploying daemon osd.2 on smithi179 2024-04-28T12:43:04.748 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:04 smithi179 ceph-mon[63029]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:04.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:04 smithi084 ceph-mon[28081]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:04.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:04 smithi161 ceph-mon[62387]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:06.892 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:06 smithi179 ceph-mon[63029]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:06 smithi084 ceph-mon[28081]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:06.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:06 smithi161 ceph-mon[62387]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:08.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:08.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:08.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:08.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:08 smithi179 ceph-mon[63029]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:08.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:08.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:08.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:08.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:08 smithi161 ceph-mon[62387]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:08.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:08.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:08.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:08.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:08 smithi084 ceph-mon[28081]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:10.775 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:10 smithi179 ceph-mon[63029]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:10.775 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:10 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:10.775 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:10 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:10.775 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:10 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:10.775 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:10 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:10.775 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:10 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:10.875 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.878+0000 7fab067fc700 1 -- 172.21.15.179:0/2794183982 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fab14109260 con 0x7fab000a3bb0 2024-04-28T12:43:10.875 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 2 on host 'smithi179' 2024-04-28T12:43:10.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.881+0000 7fab1b375700 1 -- 172.21.15.179:0/2794183982 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fab000a3bb0 msgr2=0x7fab000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:10.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.881+0000 7fab1b375700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fab000a3bb0 0x7fab000a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fab10006fd0 tx=0x7fab10008040 comp rx=0 tx=0).stop 2024-04-28T12:43:10.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.881+0000 7fab1b375700 1 -- 172.21.15.179:0/2794183982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14103580 msgr2=0x7fab1419e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:10.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.881+0000 7fab1b375700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14103580 0x7fab1419e400 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fab0800b530 tx=0x7fab0800b8f0 comp rx=0 tx=0).stop 2024-04-28T12:43:10.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.881+0000 7fab1b375700 1 -- 172.21.15.179:0/2794183982 shutdown_connections 2024-04-28T12:43:10.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.881+0000 7fab1b375700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fab000a3bb0 0x7fab000a6070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:10.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.881+0000 7fab1b375700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab14102380 0x7fab1406c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:10.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.881+0000 7fab1b375700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab14103f20 0x7fab141a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:10.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.881+0000 7fab1b375700 1 --2- 172.21.15.179:0/2794183982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14103580 0x7fab1419e400 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:10.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.881+0000 7fab1b375700 1 -- 172.21.15.179:0/2794183982 >> 172.21.15.179:0/2794183982 conn(0x7fab140fd920 msgr2=0x7fab14106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:43:10.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.881+0000 7fab1b375700 1 -- 172.21.15.179:0/2794183982 shutdown_connections 2024-04-28T12:43:10.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:43:10.881+0000 7fab1b375700 1 -- 172.21.15.179:0/2794183982 wait complete. 2024-04-28T12:43:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:10 smithi084 ceph-mon[28081]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:10.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:10 smithi161 ceph-mon[62387]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:10.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:10 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:10.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:10 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:10.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:10 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:10.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:10 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:10.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:10 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:11.511 DEBUG:teuthology.orchestra.run.smithi179:osd.2> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.2.service 2024-04-28T12:43:11.514 INFO:tasks.cephadm:Deploying osd.3 on smithi084 with /dev/vg_nvme/lv_3... 2024-04-28T12:43:11.514 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-28T12:43:11.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:11.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:11.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:11.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:11 smithi084 ceph-mon[28081]: from='osd.2 [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T12:43:11.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:11 smithi084 ceph-mon[28081]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T12:43:11.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:11 smithi084 ceph-mon[28081]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:11.947 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:43:12.071 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:12.072 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:12.072 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:12.072 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:11 smithi161 ceph-mon[62387]: from='osd.2 [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T12:43:12.072 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:11 smithi161 ceph-mon[62387]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T12:43:12.072 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:11 smithi161 ceph-mon[62387]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:12.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:12.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:12.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:12.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:11 smithi179 ceph-mon[63029]: from='osd.2 [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T12:43:12.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:11 smithi179 ceph-mon[63029]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-28T12:43:12.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:11 smithi179 ceph-mon[63029]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:12 smithi179 ceph-mon[63029]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-28T12:43:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:12 smithi179 ceph-mon[63029]: osdmap e16: 3 total, 2 up, 3 in 2024-04-28T12:43:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:12 smithi179 ceph-mon[63029]: from='osd.2 [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:43:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:12 smithi179 ceph-mon[63029]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:43:13.159 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:12 smithi084 ceph-mon[28081]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-28T12:43:13.159 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:12 smithi084 ceph-mon[28081]: osdmap e16: 3 total, 2 up, 3 in 2024-04-28T12:43:13.159 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:13.159 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:12 smithi084 ceph-mon[28081]: from='osd.2 [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:43:13.159 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:12 smithi084 ceph-mon[28081]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:43:13.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:12 smithi161 ceph-mon[62387]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-28T12:43:13.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:12 smithi161 ceph-mon[62387]: osdmap e16: 3 total, 2 up, 3 in 2024-04-28T12:43:13.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:13.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:12 smithi161 ceph-mon[62387]: from='osd.2 [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:43:13.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:12 smithi161 ceph-mon[62387]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:43:13.665 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 12:43:13 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2[68161]: 2024-04-28T12:43:13.289+0000 7f6d9fef8700 -1 osd.2 0 waiting for initial osdmap 2024-04-28T12:43:13.666 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 12:43:13 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2[68161]: 2024-04-28T12:43:13.294+0000 7f6d97ce3700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:43:14.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:13 smithi179 ceph-mon[63029]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-04-28T12:43:14.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:13 smithi179 ceph-mon[63029]: osdmap e17: 3 total, 2 up, 3 in 2024-04-28T12:43:14.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:14.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:14.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:13 smithi179 ceph-mon[63029]: from='osd.2 ' entity='osd.2' 2024-04-28T12:43:14.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:13 smithi179 ceph-mon[63029]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:14.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:13 smithi084 ceph-mon[28081]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-04-28T12:43:14.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:13 smithi084 ceph-mon[28081]: osdmap e17: 3 total, 2 up, 3 in 2024-04-28T12:43:14.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:14.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:14.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:13 smithi084 ceph-mon[28081]: from='osd.2 ' entity='osd.2' 2024-04-28T12:43:14.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:13 smithi084 ceph-mon[28081]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:13 smithi161 ceph-mon[62387]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-04-28T12:43:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:13 smithi161 ceph-mon[62387]: osdmap e17: 3 total, 2 up, 3 in 2024-04-28T12:43:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:13 smithi161 ceph-mon[62387]: from='osd.2 ' entity='osd.2' 2024-04-28T12:43:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:13 smithi161 ceph-mon[62387]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-28T12:43:14.809 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-28T12:43:14.809 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-28T12:43:14.809 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10+0 records in 2024-04-28T12:43:14.809 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2024-04-28T12:43:14.809 INFO:teuthology.orchestra.run.smithi084.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0349764 s, 300 MB/s 2024-04-28T12:43:14.810 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping successful for: 2024-04-28T12:43:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: purged_snaps scrub starts 2024-04-28T12:43:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: purged_snaps scrub ok 2024-04-28T12:43:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: osd.2 [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146] boot 2024-04-28T12:43:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: osdmap e18: 3 total, 3 up, 3 in 2024-04-28T12:43:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:14.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:14.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:14.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:14.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:14.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:14.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:14.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:15.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: purged_snaps scrub starts 2024-04-28T12:43:15.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: purged_snaps scrub ok 2024-04-28T12:43:15.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:15.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: osd.2 [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146] boot 2024-04-28T12:43:15.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: osdmap e18: 3 total, 3 up, 3 in 2024-04-28T12:43:15.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:15.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:15.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:15.104 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:15.105 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:15.105 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:15.105 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:15.105 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: purged_snaps scrub starts 2024-04-28T12:43:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: purged_snaps scrub ok 2024-04-28T12:43:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: osd.2 [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146] boot 2024-04-28T12:43:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: osdmap e18: 3 total, 3 up, 3 in 2024-04-28T12:43:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-28T12:43:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:15.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:15.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi084:vg_nvme/lv_3 2024-04-28T12:43:16.134 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:43:16.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:15 smithi179 ceph-mon[63029]: Detected new or changed devices on smithi179 2024-04-28T12:43:16.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:15 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi179 to 17088M 2024-04-28T12:43:16.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:15 smithi179 ceph-mon[63029]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:16.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:15 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-28T12:43:16.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:15 smithi084 ceph-mon[28081]: Detected new or changed devices on smithi179 2024-04-28T12:43:16.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:15 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi179 to 17088M 2024-04-28T12:43:16.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:15 smithi084 ceph-mon[28081]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:16.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:15 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-28T12:43:16.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:15 smithi161 ceph-mon[62387]: Detected new or changed devices on smithi179 2024-04-28T12:43:16.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:15 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi179 to 17088M 2024-04-28T12:43:16.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:15 smithi161 ceph-mon[62387]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:16.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:15 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-28T12:43:17.152 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:16 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-28T12:43:17.152 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:16 smithi084 ceph-mon[28081]: osdmap e19: 3 total, 3 up, 3 in 2024-04-28T12:43:17.152 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:16 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T12:43:17.152 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44531]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T12:43:17.153 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44531]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:43:17.153 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44531]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:43:17.153 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44531]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:43:17.153 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44534]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T12:43:17.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:16 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-28T12:43:17.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:16 smithi179 ceph-mon[63029]: osdmap e19: 3 total, 3 up, 3 in 2024-04-28T12:43:17.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:16 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T12:43:17.206 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:16 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-28T12:43:17.206 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:16 smithi161 ceph-mon[62387]: osdmap e19: 3 total, 3 up, 3 in 2024-04-28T12:43:17.206 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:16 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T12:43:17.403 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44534]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:43:17.403 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44534]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:43:17.403 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44534]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:43:17.492 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70653]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T12:43:17.492 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70653]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:43:17.493 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70653]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:43:17.493 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70653]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:43:17.493 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70656]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T12:43:17.493 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70656]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:43:17.493 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70656]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:43:17.493 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70656]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:43:17.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71786]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T12:43:17.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71786]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:43:17.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71786]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:43:17.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71786]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:43:17.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71789]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T12:43:17.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71789]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:43:17.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71789]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:43:17.666 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71789]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:43:17.666 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71780]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T12:43:17.666 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71780]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:43:17.666 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71780]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:43:17.666 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71780]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:43:17.666 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71783]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T12:43:17.666 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71783]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:43:17.666 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71783]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:43:17.666 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 12:43:17 smithi179 sudo[71783]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:43:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44538]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T12:43:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44538]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:43:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44538]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:43:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44538]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:43:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44541]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T12:43:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44541]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:43:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44541]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:43:17.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 sudo[44541]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:43:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70659]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-28T12:43:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70659]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:43:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70659]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:43:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70659]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:43:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70662]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-28T12:43:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70662]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-28T12:43:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70662]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-28T12:43:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 sudo[70662]: pam_unix(sudo:session): session closed for user root 2024-04-28T12:43:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-28T12:43:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: osdmap e20: 3 total, 3 up, 3 in 2024-04-28T12:43:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:18.142 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-28T12:43:18.142 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: osdmap e20: 3 total, 3 up, 3 in 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:18.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:43:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:43:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:43:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:43:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:43:18.144 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:17 smithi084 ceph-mon[28081]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:43:18.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-28T12:43:18.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: osdmap e20: 3 total, 3 up, 3 in 2024-04-28T12:43:18.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:18.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:18.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:18.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:43:18.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:43:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:17 smithi179 ceph-mon[63029]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:43:18.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-28T12:43:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-28T12:43:18.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-28T12:43:18.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:17 smithi161 ceph-mon[62387]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-28T12:43:18.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.316+0000 7f8f8e600700 1 -- 172.21.15.84:0/1743547546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f88109ac0 msgr2=0x7f8f88109ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:18.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.316+0000 7f8f8e600700 1 --2- 172.21.15.84:0/1743547546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f88109ac0 0x7f8f88109ee0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8f780098f0 tx=0x7f8f78009c00 comp rx=0 tx=0).stop 2024-04-28T12:43:18.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.317+0000 7f8f8e600700 1 -- 172.21.15.84:0/1743547546 shutdown_connections 2024-04-28T12:43:18.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.317+0000 7f8f8e600700 1 --2- 172.21.15.84:0/1743547546 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8f8810b660 0x7f8f88111f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:18.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.317+0000 7f8f8e600700 1 --2- 172.21.15.84:0/1743547546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f88109ac0 0x7f8f88109ee0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:18.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.317+0000 7f8f8e600700 1 --2- 172.21.15.84:0/1743547546 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f8810acc0 0x7f8f8810b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:18.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.317+0000 7f8f8e600700 1 -- 172.21.15.84:0/1743547546 >> 172.21.15.84:0/1743547546 conn(0x7f8f88078a90 msgr2=0x7f8f8807aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:43:18.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.317+0000 7f8f8e600700 1 -- 172.21.15.84:0/1743547546 shutdown_connections 2024-04-28T12:43:18.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.317+0000 7f8f8e600700 1 -- 172.21.15.84:0/1743547546 wait complete. 2024-04-28T12:43:18.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.318+0000 7f8f8e600700 1 Processor -- start 2024-04-28T12:43:18.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.318+0000 7f8f8e600700 1 -- start start 2024-04-28T12:43:18.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.318+0000 7f8f8e600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f88109ac0 0x7f8f8819fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:18.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.318+0000 7f8f8e600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f8810acc0 0x7f8f881a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:18.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.318+0000 7f8f8e600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8f8810b660 0x7f8f881a7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:18.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.318+0000 7f8f8e600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8f88115580 con 0x7f8f8810b660 2024-04-28T12:43:18.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.318+0000 7f8f8e600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8f88115400 con 0x7f8f88109ac0 2024-04-28T12:43:18.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.318+0000 7f8f8e600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f8f88115700 con 0x7f8f8810acc0 2024-04-28T12:43:18.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.319+0000 7f8f86a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f8810acc0 0x7f8f881a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:18.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.319+0000 7f8f87e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8f8810b660 0x7f8f881a7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:18.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.319+0000 7f8f86a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f8810acc0 0x7f8f881a01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:33048/0 (socket says 172.21.15.84:33048) 2024-04-28T12:43:18.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.319+0000 7f8f86a00700 1 -- 172.21.15.84:0/2127884760 learned_addr learned my addr 172.21.15.84:0/2127884760 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:43:18.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.319+0000 7f8f87400700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f88109ac0 0x7f8f8819fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:18.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.319+0000 7f8f86a00700 1 -- 172.21.15.84:0/2127884760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f88109ac0 msgr2=0x7f8f8819fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:18.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.319+0000 7f8f86a00700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f88109ac0 0x7f8f8819fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:18.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.319+0000 7f8f86a00700 1 -- 172.21.15.84:0/2127884760 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8f8810b660 msgr2=0x7f8f881a7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:18.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.319+0000 7f8f86a00700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8f8810b660 0x7f8f881a7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:18.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.319+0000 7f8f86a00700 1 -- 172.21.15.84:0/2127884760 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f881a78e0 con 0x7f8f8810acc0 2024-04-28T12:43:18.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.319+0000 7f8f87400700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f88109ac0 0x7f8f8819fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:43:18.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.319+0000 7f8f86a00700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f8810acc0 0x7f8f881a01f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400eb30 tx=0x7f8f7400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:43:18.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.320+0000 7f8f7fe00700 1 -- 172.21.15.84:0/2127884760 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7400cbd0 con 0x7f8f8810acc0 2024-04-28T12:43:18.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.320+0000 7f8f7fe00700 1 -- 172.21.15.84:0/2127884760 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f740044d0 con 0x7f8f8810acc0 2024-04-28T12:43:18.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.320+0000 7f8f7fe00700 1 -- 172.21.15.84:0/2127884760 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f74003e30 con 0x7f8f8810acc0 2024-04-28T12:43:18.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.320+0000 7f8f8e600700 1 -- 172.21.15.84:0/2127884760 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f881a7bd0 con 0x7f8f8810acc0 2024-04-28T12:43:18.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.320+0000 7f8f8e600700 1 -- 172.21.15.84:0/2127884760 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f881a80c0 con 0x7f8f8810acc0 2024-04-28T12:43:18.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.321+0000 7f8f8e600700 1 -- 172.21.15.84:0/2127884760 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f8806a110 con 0x7f8f8810acc0 2024-04-28T12:43:18.320 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.324+0000 7f8f7fe00700 1 -- 172.21.15.84:0/2127884760 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f8f74004000 con 0x7f8f8810acc0 2024-04-28T12:43:18.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.325+0000 7f8f7fe00700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8f6c0a3bb0 0x7f8f6c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:18.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.325+0000 7f8f7fe00700 1 -- 172.21.15.84:0/2127884760 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f8f740cf300 con 0x7f8f8810acc0 2024-04-28T12:43:18.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.325+0000 7f8f87400700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8f6c0a3bb0 0x7f8f6c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:18.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.326+0000 7f8f87400700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8f6c0a3bb0 0x7f8f6c0a6070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8f8807ae80 tx=0x7f8f78019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:43:18.326 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.329+0000 7f8f7fe00700 1 -- 172.21.15.84:0/2127884760 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8f740042f0 con 0x7f8f8810acc0 2024-04-28T12:43:18.383 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.384+0000 7f8f7fe00700 1 -- 172.21.15.84:0/2127884760 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8f74099060 con 0x7f8f8810acc0 2024-04-28T12:43:18.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:18.577+0000 7f8f8e600700 1 -- 172.21.15.84:0/2127884760 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8f88063860 con 0x7f8f6c0a3bb0 2024-04-28T12:43:19.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:18 smithi179 ceph-mon[63029]: osdmap e21: 3 total, 3 up, 3 in 2024-04-28T12:43:19.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:18 smithi179 ceph-mon[63029]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-28T12:43:19.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:18 smithi179 ceph-mon[63029]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:43:19.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:43:19.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:43:19.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:19.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:18 smithi084 ceph-mon[28081]: osdmap e21: 3 total, 3 up, 3 in 2024-04-28T12:43:19.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:18 smithi084 ceph-mon[28081]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-28T12:43:19.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:18 smithi084 ceph-mon[28081]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:43:19.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:43:19.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:43:19.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:19.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:18 smithi161 ceph-mon[62387]: osdmap e21: 3 total, 3 up, 3 in 2024-04-28T12:43:19.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:18 smithi161 ceph-mon[62387]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-28T12:43:19.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:18 smithi161 ceph-mon[62387]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:43:19.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:43:19.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:43:19.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:20.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:19 smithi179 ceph-mon[63029]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:20.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:19 smithi084 ceph-mon[28081]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:20.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:19 smithi161 ceph-mon[62387]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:22 smithi179 ceph-mon[63029]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:22.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:22 smithi084 ceph-mon[28081]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:22.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:22 smithi161 ceph-mon[62387]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:23.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:23 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4074721390' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "547c4122-2dd9-4a63-a484-f615f99b428c"}]: dispatch 2024-04-28T12:43:23.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:23 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4074721390' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "547c4122-2dd9-4a63-a484-f615f99b428c"}]': finished 2024-04-28T12:43:23.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:23 smithi179 ceph-mon[63029]: osdmap e22: 4 total, 3 up, 4 in 2024-04-28T12:43:23.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:23 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:23.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:23 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4085652973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:43:23.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:23 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4074721390' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "547c4122-2dd9-4a63-a484-f615f99b428c"}]: dispatch 2024-04-28T12:43:23.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:23 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4074721390' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "547c4122-2dd9-4a63-a484-f615f99b428c"}]': finished 2024-04-28T12:43:23.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:23 smithi084 ceph-mon[28081]: osdmap e22: 4 total, 3 up, 4 in 2024-04-28T12:43:23.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:23 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:23.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:23 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4085652973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:43:23.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:23 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4074721390' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "547c4122-2dd9-4a63-a484-f615f99b428c"}]: dispatch 2024-04-28T12:43:23.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:23 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4074721390' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "547c4122-2dd9-4a63-a484-f615f99b428c"}]': finished 2024-04-28T12:43:23.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:23 smithi161 ceph-mon[62387]: osdmap e22: 4 total, 3 up, 4 in 2024-04-28T12:43:23.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:23 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:23.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:23 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4085652973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:43:24.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:24 smithi179 ceph-mon[63029]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:24.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:24 smithi084 ceph-mon[28081]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:24.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:24 smithi161 ceph-mon[62387]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:26.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:26 smithi084 ceph-mon[28081]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:26.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:26 smithi179 ceph-mon[63029]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:26.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:26 smithi161 ceph-mon[62387]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:27.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:27 smithi179 ceph-mon[63029]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:27.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:27 smithi084 ceph-mon[28081]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:27.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:27 smithi161 ceph-mon[62387]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:30.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:30 smithi179 ceph-mon[63029]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:30.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:30 smithi084 ceph-mon[28081]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:30.972 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:30 smithi161 ceph-mon[62387]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:32.905 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:32 smithi084 ceph-mon[28081]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:32.905 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-28T12:43:32.905 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:32 smithi179 ceph-mon[63029]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-28T12:43:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:32.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:32 smithi161 ceph-mon[62387]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:32.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-28T12:43:32.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:33.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:33 smithi084 ceph-mon[28081]: Deploying daemon osd.3 on smithi084 2024-04-28T12:43:33.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:33 smithi179 ceph-mon[63029]: Deploying daemon osd.3 on smithi084 2024-04-28T12:43:33.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:33 smithi161 ceph-mon[62387]: Deploying daemon osd.3 on smithi084 2024-04-28T12:43:34.832 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:34 smithi084 ceph-mon[28081]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:34 smithi179 ceph-mon[63029]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:34.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:34 smithi161 ceph-mon[62387]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:36.868 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:36 smithi084 ceph-mon[28081]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:36 smithi179 ceph-mon[63029]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:36.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:36 smithi161 ceph-mon[62387]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:38 smithi179 ceph-mon[63029]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:38 smithi084 ceph-mon[28081]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:38.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:38 smithi161 ceph-mon[62387]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:40 smithi179 ceph-mon[63029]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:40.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:40 smithi084 ceph-mon[28081]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:40.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:40 smithi161 ceph-mon[62387]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:41 smithi084 ceph-mon[28081]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:41.964 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:41.965 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:41.965 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:41.965 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:41 smithi161 ceph-mon[62387]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:42.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:42.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:42.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:42.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:41 smithi179 ceph-mon[63029]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:44.837 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:44 smithi084 ceph-mon[28081]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:44.837 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:44.837 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:44.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:44.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:44.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:44.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:44 smithi179 ceph-mon[63029]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:44.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:44.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:44.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:44.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:44.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:44.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:44 smithi161 ceph-mon[62387]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:44.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:44.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:44.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:44.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:44.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:45.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.720+0000 7f8f7fe00700 1 -- 172.21.15.84:0/2127884760 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8f88063860 con 0x7f8f6c0a3bb0 2024-04-28T12:43:45.717 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 3 on host 'smithi084' 2024-04-28T12:43:45.720 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.724+0000 7f8f8e600700 1 -- 172.21.15.84:0/2127884760 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8f6c0a3bb0 msgr2=0x7f8f6c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:45.720 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.724+0000 7f8f8e600700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8f6c0a3bb0 0x7f8f6c0a6070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8f8807ae80 tx=0x7f8f78019040 comp rx=0 tx=0).stop 2024-04-28T12:43:45.720 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.724+0000 7f8f8e600700 1 -- 172.21.15.84:0/2127884760 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f8810acc0 msgr2=0x7f8f881a01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:45.720 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.724+0000 7f8f8e600700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f8810acc0 0x7f8f881a01f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400eb30 tx=0x7f8f7400ee40 comp rx=0 tx=0).stop 2024-04-28T12:43:45.720 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.724+0000 7f8f8e600700 1 -- 172.21.15.84:0/2127884760 shutdown_connections 2024-04-28T12:43:45.720 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.724+0000 7f8f8e600700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8f6c0a3bb0 0x7f8f6c0a6070 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:45.720 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.724+0000 7f8f8e600700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f88109ac0 0x7f8f8819fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:45.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.724+0000 7f8f8e600700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8f8810b660 0x7f8f881a7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:45.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.724+0000 7f8f8e600700 1 --2- 172.21.15.84:0/2127884760 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f8810acc0 0x7f8f881a01f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:45.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.724+0000 7f8f8e600700 1 -- 172.21.15.84:0/2127884760 >> 172.21.15.84:0/2127884760 conn(0x7f8f88078a90 msgr2=0x7f8f8810e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:43:45.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.724+0000 7f8f8e600700 1 -- 172.21.15.84:0/2127884760 shutdown_connections 2024-04-28T12:43:45.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:43:45.724+0000 7f8f8e600700 1 -- 172.21.15.84:0/2127884760 wait complete. 2024-04-28T12:43:45.888 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:45 smithi179 ceph-mon[63029]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-28T12:43:45.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:45 smithi084 ceph-mon[28081]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-28T12:43:45.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:45 smithi161 ceph-mon[62387]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-28T12:43:46.540 DEBUG:teuthology.orchestra.run.smithi084:osd.3> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.3.service 2024-04-28T12:43:46.545 INFO:tasks.cephadm:Deploying osd.4 on smithi161 with /dev/vg_nvme/lv_3... 2024-04-28T12:43:46.545 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-28T12:43:46.791 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:43:46.913 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:46 smithi161 ceph-mon[62387]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:46.913 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:46 smithi161 ceph-mon[62387]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-28T12:43:46.913 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:46 smithi161 ceph-mon[62387]: osdmap e23: 4 total, 3 up, 4 in 2024-04-28T12:43:46.914 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:46 smithi161 ceph-mon[62387]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-04-28T12:43:46.914 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:46.914 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:46.914 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:46.914 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:46 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:46 smithi179 ceph-mon[63029]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:46 smithi179 ceph-mon[63029]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-28T12:43:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:46 smithi179 ceph-mon[63029]: osdmap e23: 4 total, 3 up, 4 in 2024-04-28T12:43:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:46 smithi179 ceph-mon[63029]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-04-28T12:43:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:46 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:46 smithi084 ceph-mon[28081]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:46 smithi084 ceph-mon[28081]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-28T12:43:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:46 smithi084 ceph-mon[28081]: osdmap e23: 4 total, 3 up, 4 in 2024-04-28T12:43:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:46 smithi084 ceph-mon[28081]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-04-28T12:43:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:46.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:46 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:43:47.417 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 12:43:47 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3[46847]: 2024-04-28T12:43:47.144+0000 7f3626000700 -1 osd.3 0 waiting for initial osdmap 2024-04-28T12:43:47.417 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 12:43:47 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3[46847]: 2024-04-28T12:43:47.162+0000 7f361be00700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:43:47.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:47 smithi179 ceph-mon[63029]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-04-28T12:43:47.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:47 smithi179 ceph-mon[63029]: osdmap e24: 4 total, 3 up, 4 in 2024-04-28T12:43:47.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:47.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:47.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:47 smithi179 ceph-mon[63029]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' 2024-04-28T12:43:47.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:47 smithi084 ceph-mon[28081]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-04-28T12:43:47.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:47 smithi084 ceph-mon[28081]: osdmap e24: 4 total, 3 up, 4 in 2024-04-28T12:43:47.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:47.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:47.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:47 smithi084 ceph-mon[28081]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' 2024-04-28T12:43:47.942 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:47 smithi161 ceph-mon[62387]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-04-28T12:43:47.942 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:47 smithi161 ceph-mon[62387]: osdmap e24: 4 total, 3 up, 4 in 2024-04-28T12:43:47.943 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:47.943 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:47.943 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:47 smithi161 ceph-mon[62387]: from='osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628]' entity='osd.3' 2024-04-28T12:43:48.901 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:48 smithi161 ceph-mon[62387]: purged_snaps scrub starts 2024-04-28T12:43:48.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:48 smithi161 ceph-mon[62387]: purged_snaps scrub ok 2024-04-28T12:43:48.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:48 smithi161 ceph-mon[62387]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:48.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:48 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:48.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:48 smithi161 ceph-mon[62387]: osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628] boot 2024-04-28T12:43:48.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:48 smithi161 ceph-mon[62387]: osdmap e25: 4 total, 4 up, 4 in 2024-04-28T12:43:48.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:48 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:48.902 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-28T12:43:48.902 INFO:teuthology.orchestra.run.smithi161.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-28T12:43:48.902 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10+0 records in 2024-04-28T12:43:48.902 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2024-04-28T12:43:48.902 INFO:teuthology.orchestra.run.smithi161.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112807 s, 930 MB/s 2024-04-28T12:43:48.903 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping successful for: 2024-04-28T12:43:48.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:48 smithi179 ceph-mon[63029]: purged_snaps scrub starts 2024-04-28T12:43:48.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:48 smithi179 ceph-mon[63029]: purged_snaps scrub ok 2024-04-28T12:43:48.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:48 smithi179 ceph-mon[63029]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:48.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:48 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:48.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:48 smithi179 ceph-mon[63029]: osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628] boot 2024-04-28T12:43:48.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:48 smithi179 ceph-mon[63029]: osdmap e25: 4 total, 4 up, 4 in 2024-04-28T12:43:48.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:48 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:48.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:48 smithi084 ceph-mon[28081]: purged_snaps scrub starts 2024-04-28T12:43:48.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:48 smithi084 ceph-mon[28081]: purged_snaps scrub ok 2024-04-28T12:43:48.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:48 smithi084 ceph-mon[28081]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-28T12:43:48.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:48 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:48.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:48 smithi084 ceph-mon[28081]: osd.3 [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628] boot 2024-04-28T12:43:48.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:48 smithi084 ceph-mon[28081]: osdmap e25: 4 total, 4 up, 4 in 2024-04-28T12:43:48.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:48 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-28T12:43:49.516 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi161:vg_nvme/lv_3 2024-04-28T12:43:49.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:49 smithi161 ceph-mon[62387]: osdmap e26: 4 total, 4 up, 4 in 2024-04-28T12:43:49.763 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:43:49.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:49 smithi084 ceph-mon[28081]: osdmap e26: 4 total, 4 up, 4 in 2024-04-28T12:43:49.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:49 smithi179 ceph-mon[63029]: osdmap e26: 4 total, 4 up, 4 in 2024-04-28T12:43:50.850 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:50 smithi161 ceph-mon[62387]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:43:50.850 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:50 smithi161 ceph-mon[62387]: osdmap e27: 4 total, 4 up, 4 in 2024-04-28T12:43:50.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:50 smithi179 ceph-mon[63029]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:43:50.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:50 smithi179 ceph-mon[63029]: osdmap e27: 4 total, 4 up, 4 in 2024-04-28T12:43:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:50 smithi084 ceph-mon[28081]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:43:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:50 smithi084 ceph-mon[28081]: osdmap e27: 4 total, 4 up, 4 in 2024-04-28T12:43:51.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.376+0000 7ff878309700 1 -- 172.21.15.161:0/3407748564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8701036d0 msgr2=0x7ff870109f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:51.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.376+0000 7ff878309700 1 --2- 172.21.15.161:0/3407748564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8701036d0 0x7ff870109f80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff86800b0d0 tx=0x7ff86800b3e0 comp rx=0 tx=0).stop 2024-04-28T12:43:51.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 -- 172.21.15.161:0/3407748564 shutdown_connections 2024-04-28T12:43:51.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 --2- 172.21.15.161:0/3407748564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8701036d0 0x7ff870109f80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 --2- 172.21.15.161:0/3407748564 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff870101b30 0x7ff870101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 --2- 172.21.15.161:0/3407748564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff870102d30 0x7ff870103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 -- 172.21.15.161:0/3407748564 >> 172.21.15.161:0/3407748564 conn(0x7ff8700fd0d0 msgr2=0x7ff8700ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 -- 172.21.15.161:0/3407748564 shutdown_connections 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 -- 172.21.15.161:0/3407748564 wait complete. 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 Processor -- start 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 -- start start 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff870101b30 0x7ff87010b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff870102d30 0x7ff87019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8701036d0 0x7ff8701a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff87010df40 con 0x7ff870102d30 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff87010ddc0 con 0x7ff8701036d0 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.377+0000 7ff878309700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff87010e0c0 con 0x7ff870101b30 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8768a6700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8701036d0 0x7ff8701a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8768a6700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8701036d0 0x7ff8701a0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35696/0 (socket says 172.21.15.161:35696) 2024-04-28T12:43:51.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8768a6700 1 -- 172.21.15.161:0/2945584253 learned_addr learned my addr 172.21.15.161:0/2945584253 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-28T12:43:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8760a5700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff870101b30 0x7ff87010b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8768a6700 1 -- 172.21.15.161:0/2945584253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff870101b30 msgr2=0x7ff87010b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8768a6700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff870101b30 0x7ff87010b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8768a6700 1 -- 172.21.15.161:0/2945584253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff870102d30 msgr2=0x7ff87019e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:43:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8768a6700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff870102d30 0x7ff87019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:43:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8768a6700 1 -- 172.21.15.161:0/2945584253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8701a1310 con 0x7ff8701036d0 2024-04-28T12:43:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8768a6700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8701036d0 0x7ff8701a0a40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff868009fd0 tx=0x7ff868004e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:43:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8677fe700 1 -- 172.21.15.161:0/2945584253 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86801f030 con 0x7ff8701036d0 2024-04-28T12:43:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8677fe700 1 -- 172.21.15.161:0/2945584253 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff868007a80 con 0x7ff8701036d0 2024-04-28T12:43:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff878309700 1 -- 172.21.15.161:0/2945584253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8701a15a0 con 0x7ff8701036d0 2024-04-28T12:43:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff878309700 1 -- 172.21.15.161:0/2945584253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8701a1a70 con 0x7ff8701036d0 2024-04-28T12:43:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.378+0000 7ff8677fe700 1 -- 172.21.15.161:0/2945584253 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868004810 con 0x7ff8701036d0 2024-04-28T12:43:51.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.379+0000 7ff878309700 1 -- 172.21.15.161:0/2945584253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff87002e490 con 0x7ff8701036d0 2024-04-28T12:43:51.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.380+0000 7ff8677fe700 1 -- 172.21.15.161:0/2945584253 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff868003a00 con 0x7ff8701036d0 2024-04-28T12:43:51.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.381+0000 7ff8677fe700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff85c0a3c40 0x7ff85c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:43:51.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.381+0000 7ff8677fe700 1 -- 172.21.15.161:0/2945584253 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7ff86800f020 con 0x7ff8701036d0 2024-04-28T12:43:51.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.381+0000 7ff8760a5700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff85c0a3c40 0x7ff85c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:43:51.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.382+0000 7ff8760a5700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff85c0a3c40 0x7ff85c0a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff870102b90 tx=0x7ff86c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:43:51.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.386+0000 7ff8677fe700 1 -- 172.21.15.161:0/2945584253 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff868029020 con 0x7ff8701036d0 2024-04-28T12:43:51.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:43:51.593+0000 7ff878309700 1 -- 172.21.15.161:0/2945584253 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff870063860 con 0x7ff85c0a3c40 2024-04-28T12:43:51.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:51 smithi179 ceph-mon[63029]: osdmap e28: 4 total, 4 up, 4 in 2024-04-28T12:43:51.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:51 smithi179 ceph-mon[63029]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:43:51.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:51 smithi179 ceph-mon[63029]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:43:51.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:51 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:43:51.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:51 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:43:51.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:51 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:51.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:51 smithi084 ceph-mon[28081]: osdmap e28: 4 total, 4 up, 4 in 2024-04-28T12:43:51.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:51 smithi084 ceph-mon[28081]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:43:51.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:51 smithi084 ceph-mon[28081]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:43:51.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:51 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:43:51.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:51 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:43:51.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:51 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:51.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:51 smithi161 ceph-mon[62387]: osdmap e28: 4 total, 4 up, 4 in 2024-04-28T12:43:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:51 smithi161 ceph-mon[62387]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:43:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:51 smithi161 ceph-mon[62387]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:43:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:51 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:43:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:51 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:43:51.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:51 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:52.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:52 smithi161 ceph-mon[62387]: osdmap e29: 4 total, 4 up, 4 in 2024-04-28T12:43:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:52 smithi179 ceph-mon[63029]: osdmap e29: 4 total, 4 up, 4 in 2024-04-28T12:43:52.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:52 smithi084 ceph-mon[28081]: osdmap e29: 4 total, 4 up, 4 in 2024-04-28T12:43:54.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:53 smithi179 ceph-mon[63029]: Detected new or changed devices on smithi084 2024-04-28T12:43:54.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:53 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:54.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:53 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:54.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:53 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:54.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:53 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:54.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:53 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi084 to 8544M 2024-04-28T12:43:54.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:53 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:54.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:53 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:54.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:53 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:54.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:53 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:54.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:53 smithi179 ceph-mon[63029]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-28T12:43:54.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:53 smithi084 ceph-mon[28081]: Detected new or changed devices on smithi084 2024-04-28T12:43:54.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:53 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:54.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:53 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:54.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:53 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:54.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:53 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:54.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:53 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi084 to 8544M 2024-04-28T12:43:54.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:53 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:54.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:53 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:54.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:53 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:54.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:53 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:54.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:53 smithi084 ceph-mon[28081]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-28T12:43:54.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:53 smithi161 ceph-mon[62387]: Detected new or changed devices on smithi084 2024-04-28T12:43:54.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:53 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:54.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:53 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:54.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:53 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:54.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:53 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:43:54.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:53 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi084 to 8544M 2024-04-28T12:43:54.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:53 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:54.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:53 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:43:54.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:53 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:43:54.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:53 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:43:54.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:53 smithi161 ceph-mon[62387]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-28T12:43:54.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:54 smithi161 ceph-mon[62387]: from='client.? 172.21.15.161:0/2258321045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd99b566-f29a-42c6-b04d-7122b6787464"}]: dispatch 2024-04-28T12:43:54.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:54 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd99b566-f29a-42c6-b04d-7122b6787464"}]: dispatch 2024-04-28T12:43:54.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:54 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd99b566-f29a-42c6-b04d-7122b6787464"}]': finished 2024-04-28T12:43:54.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:54 smithi161 ceph-mon[62387]: osdmap e30: 5 total, 4 up, 5 in 2024-04-28T12:43:54.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:54 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:43:55.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:54 smithi179 ceph-mon[63029]: from='client.? 172.21.15.161:0/2258321045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd99b566-f29a-42c6-b04d-7122b6787464"}]: dispatch 2024-04-28T12:43:55.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:54 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd99b566-f29a-42c6-b04d-7122b6787464"}]: dispatch 2024-04-28T12:43:55.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:54 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd99b566-f29a-42c6-b04d-7122b6787464"}]': finished 2024-04-28T12:43:55.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:54 smithi179 ceph-mon[63029]: osdmap e30: 5 total, 4 up, 5 in 2024-04-28T12:43:55.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:54 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:43:55.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:54 smithi084 ceph-mon[28081]: from='client.? 172.21.15.161:0/2258321045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd99b566-f29a-42c6-b04d-7122b6787464"}]: dispatch 2024-04-28T12:43:55.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:54 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd99b566-f29a-42c6-b04d-7122b6787464"}]: dispatch 2024-04-28T12:43:55.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:54 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd99b566-f29a-42c6-b04d-7122b6787464"}]': finished 2024-04-28T12:43:55.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:54 smithi084 ceph-mon[28081]: osdmap e30: 5 total, 4 up, 5 in 2024-04-28T12:43:55.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:54 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:43:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:55 smithi179 ceph-mon[63029]: from='client.? 172.21.15.161:0/1108595898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:43:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:55 smithi179 ceph-mon[63029]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-28T12:43:56.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:55 smithi084 ceph-mon[28081]: from='client.? 172.21.15.161:0/1108595898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:43:56.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:55 smithi084 ceph-mon[28081]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-28T12:43:56.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:55 smithi161 ceph-mon[62387]: from='client.? 172.21.15.161:0/1108595898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:43:56.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:55 smithi161 ceph-mon[62387]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-28T12:43:58.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:43:58 smithi179 ceph-mon[63029]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-28T12:43:58.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:43:58 smithi084 ceph-mon[28081]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-28T12:43:58.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:43:58 smithi161 ceph-mon[62387]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-28T12:44:00.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:00 smithi161 ceph-mon[62387]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:00 smithi179 ceph-mon[63029]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:00.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:00 smithi084 ceph-mon[28081]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:01.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:01 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-28T12:44:01.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:01 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:01.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:01 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-28T12:44:01.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:01 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:01.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:01 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-28T12:44:01.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:01 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:02.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:02 smithi161 ceph-mon[62387]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:02.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:02 smithi161 ceph-mon[62387]: Deploying daemon osd.4 on smithi161 2024-04-28T12:44:02.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:02 smithi179 ceph-mon[63029]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:02.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:02 smithi179 ceph-mon[63029]: Deploying daemon osd.4 on smithi161 2024-04-28T12:44:02.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:02 smithi084 ceph-mon[28081]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:02.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:02 smithi084 ceph-mon[28081]: Deploying daemon osd.4 on smithi161 2024-04-28T12:44:04.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:04 smithi161 ceph-mon[62387]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:04.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:04 smithi179 ceph-mon[63029]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:04.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:04 smithi084 ceph-mon[28081]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:06.867 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:06 smithi161 ceph-mon[62387]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:06.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:06 smithi179 ceph-mon[63029]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:06.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:06 smithi084 ceph-mon[28081]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:08.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:08.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:08.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:08.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:08 smithi179 ceph-mon[63029]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:08.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:08.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:08.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:08.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:08 smithi084 ceph-mon[28081]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:08.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:08.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:08.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:08.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:08 smithi161 ceph-mon[62387]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:10.856 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:10 smithi161 ceph-mon[62387]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:10.857 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:10 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:10.857 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:10 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:10.857 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:10 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:10.857 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:10 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:10.857 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:10 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:10.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:10 smithi179 ceph-mon[63029]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:10.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:10 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:10.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:10 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:10.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:10 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:10.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:10 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:10.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:10 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:10 smithi084 ceph-mon[28081]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:10 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:10.927 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 4 on host 'smithi161' 2024-04-28T12:44:10.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.931+0000 7ff8677fe700 1 -- 172.21.15.161:0/2945584253 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff870063860 con 0x7ff85c0a3c40 2024-04-28T12:44:10.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.933+0000 7ff878309700 1 -- 172.21.15.161:0/2945584253 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff85c0a3c40 msgr2=0x7ff85c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:10.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.933+0000 7ff878309700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff85c0a3c40 0x7ff85c0a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff870102b90 tx=0x7ff86c009190 comp rx=0 tx=0).stop 2024-04-28T12:44:10.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.933+0000 7ff878309700 1 -- 172.21.15.161:0/2945584253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8701036d0 msgr2=0x7ff8701a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:10.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.933+0000 7ff878309700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8701036d0 0x7ff8701a0a40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff868009fd0 tx=0x7ff868004e40 comp rx=0 tx=0).stop 2024-04-28T12:44:10.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.934+0000 7ff878309700 1 -- 172.21.15.161:0/2945584253 shutdown_connections 2024-04-28T12:44:10.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.934+0000 7ff878309700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff85c0a3c40 0x7ff85c0a6100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:10.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.934+0000 7ff878309700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8701036d0 0x7ff8701a0a40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:10.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.934+0000 7ff878309700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff870101b30 0x7ff87010b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:10.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.934+0000 7ff878309700 1 --2- 172.21.15.161:0/2945584253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff870102d30 0x7ff87019e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:10.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.934+0000 7ff878309700 1 -- 172.21.15.161:0/2945584253 >> 172.21.15.161:0/2945584253 conn(0x7ff8700fd0d0 msgr2=0x7ff870106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:44:10.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.934+0000 7ff878309700 1 -- 172.21.15.161:0/2945584253 shutdown_connections 2024-04-28T12:44:10.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:44:10.934+0000 7ff878309700 1 -- 172.21.15.161:0/2945584253 wait complete. 2024-04-28T12:44:11.623 DEBUG:teuthology.orchestra.run.smithi161:osd.4> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.4.service 2024-04-28T12:44:11.626 INFO:tasks.cephadm:Deploying osd.5 on smithi179 with /dev/vg_nvme/lv_3... 2024-04-28T12:44:11.626 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-28T12:44:11.870 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:44:11.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:11.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:11.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:11.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:11 smithi179 ceph-mon[63029]: from='osd.4 [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T12:44:11.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:11 smithi179 ceph-mon[63029]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T12:44:11.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:11 smithi179 ceph-mon[63029]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:12.155 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:12.155 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:12.156 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:12.156 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:11 smithi161 ceph-mon[62387]: from='osd.4 [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T12:44:12.156 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:11 smithi161 ceph-mon[62387]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T12:44:12.156 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:11 smithi161 ceph-mon[62387]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:11 smithi084 ceph-mon[28081]: from='osd.4 [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T12:44:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:11 smithi084 ceph-mon[28081]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-28T12:44:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:11 smithi084 ceph-mon[28081]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:13.168 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:12 smithi161 ceph-mon[62387]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-28T12:44:13.168 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:12 smithi161 ceph-mon[62387]: from='osd.4 [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:44:13.168 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:12 smithi161 ceph-mon[62387]: osdmap e31: 5 total, 4 up, 5 in 2024-04-28T12:44:13.168 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:13.168 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:12 smithi161 ceph-mon[62387]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:44:13.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:12 smithi179 ceph-mon[63029]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-28T12:44:13.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:12 smithi179 ceph-mon[63029]: from='osd.4 [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:44:13.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:12 smithi179 ceph-mon[63029]: osdmap e31: 5 total, 4 up, 5 in 2024-04-28T12:44:13.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:13.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:12 smithi179 ceph-mon[63029]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:44:13.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:12 smithi084 ceph-mon[28081]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-28T12:44:13.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:12 smithi084 ceph-mon[28081]: from='osd.4 [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:44:13.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:12 smithi084 ceph-mon[28081]: osdmap e31: 5 total, 4 up, 5 in 2024-04-28T12:44:13.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:13.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:12 smithi084 ceph-mon[28081]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:44:13.492 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 12:44:13 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4[73486]: 2024-04-28T12:44:13.351+0000 7fd13cc9e700 -1 osd.4 0 waiting for initial osdmap 2024-04-28T12:44:13.493 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 12:44:13 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4[73486]: 2024-04-28T12:44:13.358+0000 7fd134a89700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:44:14.144 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-28T12:44:14.144 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-28T12:44:14.144 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-04-28T12:44:14.145 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-04-28T12:44:14.145 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113004 s, 928 MB/s 2024-04-28T12:44:14.145 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-04-28T12:44:14.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:13 smithi179 ceph-mon[63029]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-04-28T12:44:14.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:13 smithi179 ceph-mon[63029]: osdmap e32: 5 total, 4 up, 5 in 2024-04-28T12:44:14.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:14.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:14.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:13 smithi179 ceph-mon[63029]: from='osd.4 ' entity='osd.4' 2024-04-28T12:44:14.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:13 smithi179 ceph-mon[63029]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:14.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:13 smithi161 ceph-mon[62387]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-04-28T12:44:14.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:13 smithi161 ceph-mon[62387]: osdmap e32: 5 total, 4 up, 5 in 2024-04-28T12:44:14.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:13 smithi161 ceph-mon[62387]: from='osd.4 ' entity='osd.4' 2024-04-28T12:44:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:13 smithi161 ceph-mon[62387]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:14.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:13 smithi084 ceph-mon[28081]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-04-28T12:44:14.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:13 smithi084 ceph-mon[28081]: osdmap e32: 5 total, 4 up, 5 in 2024-04-28T12:44:14.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:14.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:14.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:13 smithi084 ceph-mon[28081]: from='osd.4 ' entity='osd.4' 2024-04-28T12:44:14.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:13 smithi084 ceph-mon[28081]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-28T12:44:14.809 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi179:vg_nvme/lv_3 2024-04-28T12:44:15.049 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:44:15.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: purged_snaps scrub starts 2024-04-28T12:44:15.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: purged_snaps scrub ok 2024-04-28T12:44:15.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:15.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: osd.4 [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515] boot 2024-04-28T12:44:15.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: osdmap e33: 5 total, 5 up, 5 in 2024-04-28T12:44:15.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:15.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:15.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:15.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:15.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:15.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:15.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:15.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:15.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: purged_snaps scrub starts 2024-04-28T12:44:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: purged_snaps scrub ok 2024-04-28T12:44:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: osd.4 [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515] boot 2024-04-28T12:44:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: osdmap e33: 5 total, 5 up, 5 in 2024-04-28T12:44:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:15.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:15.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:15.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:15.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: purged_snaps scrub starts 2024-04-28T12:44:15.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: purged_snaps scrub ok 2024-04-28T12:44:15.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:15.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: osd.4 [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515] boot 2024-04-28T12:44:15.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: osdmap e33: 5 total, 5 up, 5 in 2024-04-28T12:44:15.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-28T12:44:15.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:15.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:15.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:15.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:15.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:15.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:15.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:15.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:16.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:15 smithi179 ceph-mon[63029]: Detected new or changed devices on smithi161 2024-04-28T12:44:16.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:15 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi161 to 8544M 2024-04-28T12:44:16.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:15 smithi179 ceph-mon[63029]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:16.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:15 smithi161 ceph-mon[62387]: Detected new or changed devices on smithi161 2024-04-28T12:44:16.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:15 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi161 to 8544M 2024-04-28T12:44:16.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:15 smithi161 ceph-mon[62387]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:16.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:15 smithi084 ceph-mon[28081]: Detected new or changed devices on smithi161 2024-04-28T12:44:16.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:15 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi161 to 8544M 2024-04-28T12:44:16.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:15 smithi084 ceph-mon[28081]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 -- 172.21.15.179:0/1299441030 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ac0103580 msgr2=0x7f6ac01039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 --2- 172.21.15.179:0/1299441030 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ac0103580 0x7f6ac01039e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009990 tx=0x7f6ab0009ca0 comp rx=0 tx=0).stop 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 -- 172.21.15.179:0/1299441030 shutdown_connections 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 --2- 172.21.15.179:0/1299441030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ac0103f20 0x7f6ac010a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 --2- 172.21.15.179:0/1299441030 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ac0102380 0x7f6ac01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 --2- 172.21.15.179:0/1299441030 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ac0103580 0x7f6ac01039e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 -- 172.21.15.179:0/1299441030 >> 172.21.15.179:0/1299441030 conn(0x7f6ac00fd940 msgr2=0x7f6ac00ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 -- 172.21.15.179:0/1299441030 shutdown_connections 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 -- 172.21.15.179:0/1299441030 wait complete. 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 Processor -- start 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 -- start start 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ac0102380 0x7f6ac0106310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:16.664 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ac0103580 0x7f6ac0104960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ac0103f20 0x7f6ac0104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac010de90 con 0x7f6ac0102380 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac010dd10 con 0x7f6ac0103f20 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.666+0000 7f6ac83ac700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac010e010 con 0x7f6ac0103580 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ac5947700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ac0103580 0x7f6ac0104960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ac5947700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ac0103580 0x7f6ac0104960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:58510/0 (socket says 172.21.15.179:58510) 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ac5947700 1 -- 172.21.15.179:0/2842681343 learned_addr learned my addr 172.21.15.179:0/2842681343 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ac6949700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ac0103f20 0x7f6ac0104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ac5947700 1 -- 172.21.15.179:0/2842681343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ac0103f20 msgr2=0x7f6ac0104ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ac5947700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ac0103f20 0x7f6ac0104ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ac5947700 1 -- 172.21.15.179:0/2842681343 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ac0102380 msgr2=0x7f6ac0106310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ac5947700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ac0102380 0x7f6ac0106310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ac5947700 1 -- 172.21.15.179:0/2842681343 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac0105750 con 0x7f6ac0103580 2024-04-28T12:44:16.665 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ac5947700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ac0103580 0x7f6ac0104960 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0006010 tx=0x7f6ab0009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:44:16.666 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ab77fe700 1 -- 172.21.15.179:0/2842681343 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0027070 con 0x7f6ac0103580 2024-04-28T12:44:16.666 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ac83ac700 1 -- 172.21.15.179:0/2842681343 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac01a3770 con 0x7f6ac0103580 2024-04-28T12:44:16.666 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ab77fe700 1 -- 172.21.15.179:0/2842681343 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ab0022470 con 0x7f6ac0103580 2024-04-28T12:44:16.666 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ab77fe700 1 -- 172.21.15.179:0/2842681343 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab000f6e0 con 0x7f6ac0103580 2024-04-28T12:44:16.666 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.667+0000 7f6ac83ac700 1 -- 172.21.15.179:0/2842681343 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac01a3bc0 con 0x7f6ac0103580 2024-04-28T12:44:16.667 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.668+0000 7f6ac83ac700 1 -- 172.21.15.179:0/2842681343 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ac006a110 con 0x7f6ac0103580 2024-04-28T12:44:16.667 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.669+0000 7f6ab77fe700 1 -- 172.21.15.179:0/2842681343 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6ab002a070 con 0x7f6ac0103580 2024-04-28T12:44:16.667 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.670+0000 7f6ab77fe700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6aac0a3c40 0x7f6aac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:16.667 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.670+0000 7f6ab77fe700 1 -- 172.21.15.179:0/2842681343 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f6ab00d01b0 con 0x7f6ac0103580 2024-04-28T12:44:16.667 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.670+0000 7f6ac6148700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6aac0a3c40 0x7f6aac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:16.667 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.671+0000 7f6ac6148700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6aac0a3c40 0x7f6aac0a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ac01033e0 tx=0x7f6abc006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:44:16.670 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.675+0000 7f6ab77fe700 1 -- 172.21.15.179:0/2842681343 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6ab0013070 con 0x7f6ac0103580 2024-04-28T12:44:16.873 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:16.877+0000 7f6ac83ac700 1 -- 172.21.15.179:0/2842681343 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6ac00ffb70 con 0x7f6aac0a3c40 2024-04-28T12:44:17.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:16 smithi179 ceph-mon[63029]: osdmap e34: 5 total, 5 up, 5 in 2024-04-28T12:44:17.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:16 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:17.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:16 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:17.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:16 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:17.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:16 smithi161 ceph-mon[62387]: osdmap e34: 5 total, 5 up, 5 in 2024-04-28T12:44:17.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:16 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:17.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:16 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:17.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:16 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:17.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:16 smithi084 ceph-mon[28081]: osdmap e34: 5 total, 5 up, 5 in 2024-04-28T12:44:17.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:16 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:17.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:16 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:17.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:16 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:18.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:17 smithi179 ceph-mon[63029]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:18.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:17 smithi179 ceph-mon[63029]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:18.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:17 smithi161 ceph-mon[62387]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:18.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:17 smithi161 ceph-mon[62387]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:18.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:17 smithi084 ceph-mon[28081]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:18.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:17 smithi084 ceph-mon[28081]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:20 smithi179 ceph-mon[63029]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:20 smithi179 ceph-mon[63029]: from='client.? 172.21.15.179:0/4024316427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0697f1f-142e-43d7-b718-73f40970276e"}]: dispatch 2024-04-28T12:44:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:20 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0697f1f-142e-43d7-b718-73f40970276e"}]: dispatch 2024-04-28T12:44:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:20 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0697f1f-142e-43d7-b718-73f40970276e"}]': finished 2024-04-28T12:44:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:20 smithi179 ceph-mon[63029]: osdmap e35: 6 total, 5 up, 6 in 2024-04-28T12:44:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:20 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:20.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:20 smithi179 ceph-mon[63029]: from='client.? 172.21.15.179:0/918870486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:20 smithi084 ceph-mon[28081]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:20 smithi084 ceph-mon[28081]: from='client.? 172.21.15.179:0/4024316427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0697f1f-142e-43d7-b718-73f40970276e"}]: dispatch 2024-04-28T12:44:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:20 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0697f1f-142e-43d7-b718-73f40970276e"}]: dispatch 2024-04-28T12:44:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:20 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0697f1f-142e-43d7-b718-73f40970276e"}]': finished 2024-04-28T12:44:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:20 smithi084 ceph-mon[28081]: osdmap e35: 6 total, 5 up, 6 in 2024-04-28T12:44:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:20 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:20 smithi084 ceph-mon[28081]: from='client.? 172.21.15.179:0/918870486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:20.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:20 smithi161 ceph-mon[62387]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:20.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:20 smithi161 ceph-mon[62387]: from='client.? 172.21.15.179:0/4024316427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0697f1f-142e-43d7-b718-73f40970276e"}]: dispatch 2024-04-28T12:44:20.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:20 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0697f1f-142e-43d7-b718-73f40970276e"}]: dispatch 2024-04-28T12:44:20.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:20 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0697f1f-142e-43d7-b718-73f40970276e"}]': finished 2024-04-28T12:44:20.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:20 smithi161 ceph-mon[62387]: osdmap e35: 6 total, 5 up, 6 in 2024-04-28T12:44:20.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:20 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:20.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:20 smithi161 ceph-mon[62387]: from='client.? 172.21.15.179:0/918870486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:22 smithi179 ceph-mon[63029]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:22.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:22 smithi084 ceph-mon[28081]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:22.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:22 smithi161 ceph-mon[62387]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:24.826 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:24 smithi179 ceph-mon[63029]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:24.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:24 smithi084 ceph-mon[28081]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:24.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:24 smithi161 ceph-mon[62387]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:26.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:26 smithi179 ceph-mon[63029]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:26.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:26 smithi084 ceph-mon[28081]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:26.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:26 smithi161 ceph-mon[62387]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:27.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:27 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-28T12:44:27.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:27 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:27.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:27 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-28T12:44:27.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:27 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:27.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:27 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-28T12:44:27.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:27 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:28.898 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:28 smithi179 ceph-mon[63029]: Deploying daemon osd.5 on smithi179 2024-04-28T12:44:28.899 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:28 smithi179 ceph-mon[63029]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:28.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:28 smithi084 ceph-mon[28081]: Deploying daemon osd.5 on smithi179 2024-04-28T12:44:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:28 smithi084 ceph-mon[28081]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:28.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:28 smithi161 ceph-mon[62387]: Deploying daemon osd.5 on smithi179 2024-04-28T12:44:28.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:28 smithi161 ceph-mon[62387]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:30.856 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:30 smithi179 ceph-mon[63029]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:30.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:30 smithi084 ceph-mon[28081]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:30.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:30 smithi161 ceph-mon[62387]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:32 smithi179 ceph-mon[63029]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:32.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:32 smithi084 ceph-mon[28081]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:32.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:32.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:32.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:32.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:32 smithi161 ceph-mon[62387]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:32.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:32.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:32.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:34 smithi179 ceph-mon[63029]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:34.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:34 smithi084 ceph-mon[28081]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:34.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:34 smithi161 ceph-mon[62387]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:36.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.565+0000 7f6ab77fe700 1 -- 172.21.15.179:0/2842681343 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6ac00ffb70 con 0x7f6aac0a3c40 2024-04-28T12:44:36.562 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 5 on host 'smithi179' 2024-04-28T12:44:36.563 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.567+0000 7f6ac83ac700 1 -- 172.21.15.179:0/2842681343 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6aac0a3c40 msgr2=0x7f6aac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:36.563 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.567+0000 7f6ac83ac700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6aac0a3c40 0x7f6aac0a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ac01033e0 tx=0x7f6abc006d20 comp rx=0 tx=0).stop 2024-04-28T12:44:36.563 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.567+0000 7f6ac83ac700 1 -- 172.21.15.179:0/2842681343 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ac0103580 msgr2=0x7f6ac0104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:36.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.567+0000 7f6ac83ac700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ac0103580 0x7f6ac0104960 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0006010 tx=0x7f6ab0009f90 comp rx=0 tx=0).stop 2024-04-28T12:44:36.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.567+0000 7f6ac83ac700 1 -- 172.21.15.179:0/2842681343 shutdown_connections 2024-04-28T12:44:36.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.567+0000 7f6ac83ac700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6aac0a3c40 0x7f6aac0a6100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:36.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.567+0000 7f6ac83ac700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ac0103f20 0x7f6ac0104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:36.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.567+0000 7f6ac83ac700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ac0102380 0x7f6ac0106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:36.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.567+0000 7f6ac83ac700 1 --2- 172.21.15.179:0/2842681343 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ac0103580 0x7f6ac0104960 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:36.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.567+0000 7f6ac83ac700 1 -- 172.21.15.179:0/2842681343 >> 172.21.15.179:0/2842681343 conn(0x7f6ac00fd940 msgr2=0x7f6ac00ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:44:36.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.568+0000 7f6ac83ac700 1 -- 172.21.15.179:0/2842681343 shutdown_connections 2024-04-28T12:44:36.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:44:36.568+0000 7f6ac83ac700 1 -- 172.21.15.179:0/2842681343 wait complete. 2024-04-28T12:44:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:36 smithi179 ceph-mon[63029]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:36 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:36 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:36 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:36 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:36 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:36 smithi179 ceph-mon[63029]: from='osd.5 [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T12:44:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:36 smithi179 ceph-mon[63029]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T12:44:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:36 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:36 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:36.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:36 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:36 smithi084 ceph-mon[28081]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:36.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:36 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:36 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:36 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:36.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:36 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:36.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:36 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:36 smithi084 ceph-mon[28081]: from='osd.5 [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T12:44:36.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:36 smithi084 ceph-mon[28081]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T12:44:36.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:36 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:36 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:36.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:36 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:36 smithi161 ceph-mon[62387]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:36.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:36 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:36 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:36 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:36.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:36 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:36.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:36 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:36 smithi161 ceph-mon[62387]: from='osd.5 [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T12:44:36.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:36 smithi161 ceph-mon[62387]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-28T12:44:36.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:36 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:36.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:36 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:44:36.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:36 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:37.353 DEBUG:teuthology.orchestra.run.smithi179:osd.5> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.5.service 2024-04-28T12:44:37.355 INFO:tasks.cephadm:Deploying osd.6 on smithi084 with /dev/vg_nvme/lv_2... 2024-04-28T12:44:37.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-28T12:44:37.786 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:44:37.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:37 smithi179 ceph-mon[63029]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-28T12:44:37.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:37 smithi179 ceph-mon[63029]: osdmap e36: 6 total, 5 up, 6 in 2024-04-28T12:44:37.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:37.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:37 smithi179 ceph-mon[63029]: from='osd.5 [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:44:37.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:37 smithi179 ceph-mon[63029]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:44:37.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:37 smithi179 ceph-mon[63029]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:37.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:37 smithi084 ceph-mon[28081]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-28T12:44:37.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:37 smithi084 ceph-mon[28081]: osdmap e36: 6 total, 5 up, 6 in 2024-04-28T12:44:37.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:37.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:37 smithi084 ceph-mon[28081]: from='osd.5 [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:44:37.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:37 smithi084 ceph-mon[28081]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:44:37.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:37 smithi084 ceph-mon[28081]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:37.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:37 smithi161 ceph-mon[62387]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-28T12:44:37.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:37 smithi161 ceph-mon[62387]: osdmap e36: 6 total, 5 up, 6 in 2024-04-28T12:44:37.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:37.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:37 smithi161 ceph-mon[62387]: from='osd.5 [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:44:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:37 smithi161 ceph-mon[62387]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:44:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:37 smithi161 ceph-mon[62387]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:38.415 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 12:44:38 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5[74676]: 2024-04-28T12:44:38.040+0000 7faef3f2b700 -1 osd.5 0 waiting for initial osdmap 2024-04-28T12:44:38.415 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 12:44:38 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5[74676]: 2024-04-28T12:44:38.048+0000 7faeebd16700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:44:38.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:38 smithi179 ceph-mon[63029]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-04-28T12:44:38.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:38 smithi179 ceph-mon[63029]: osdmap e37: 6 total, 5 up, 6 in 2024-04-28T12:44:38.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:38.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:38.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:38 smithi179 ceph-mon[63029]: from='osd.5 ' entity='osd.5' 2024-04-28T12:44:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:38 smithi084 ceph-mon[28081]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-04-28T12:44:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:38 smithi084 ceph-mon[28081]: osdmap e37: 6 total, 5 up, 6 in 2024-04-28T12:44:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:38 smithi084 ceph-mon[28081]: from='osd.5 ' entity='osd.5' 2024-04-28T12:44:38.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:38 smithi161 ceph-mon[62387]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-04-28T12:44:38.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:38 smithi161 ceph-mon[62387]: osdmap e37: 6 total, 5 up, 6 in 2024-04-28T12:44:38.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:38.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:38.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:38 smithi161 ceph-mon[62387]: from='osd.5 ' entity='osd.5' 2024-04-28T12:44:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:39 smithi084 ceph-mon[28081]: purged_snaps scrub starts 2024-04-28T12:44:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:39 smithi084 ceph-mon[28081]: purged_snaps scrub ok 2024-04-28T12:44:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:39 smithi084 ceph-mon[28081]: osd.5 [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251] boot 2024-04-28T12:44:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:39 smithi084 ceph-mon[28081]: osdmap e38: 6 total, 6 up, 6 in 2024-04-28T12:44:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:39 smithi084 ceph-mon[28081]: osdmap e39: 6 total, 6 up, 6 in 2024-04-28T12:44:39.870 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:39 smithi084 ceph-mon[28081]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:39 smithi179 ceph-mon[63029]: purged_snaps scrub starts 2024-04-28T12:44:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:39 smithi179 ceph-mon[63029]: purged_snaps scrub ok 2024-04-28T12:44:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:39 smithi179 ceph-mon[63029]: osd.5 [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251] boot 2024-04-28T12:44:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:39 smithi179 ceph-mon[63029]: osdmap e38: 6 total, 6 up, 6 in 2024-04-28T12:44:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:39 smithi179 ceph-mon[63029]: osdmap e39: 6 total, 6 up, 6 in 2024-04-28T12:44:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:39 smithi179 ceph-mon[63029]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:39.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:39 smithi161 ceph-mon[62387]: purged_snaps scrub starts 2024-04-28T12:44:39.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:39 smithi161 ceph-mon[62387]: purged_snaps scrub ok 2024-04-28T12:44:39.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:39 smithi161 ceph-mon[62387]: osd.5 [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251] boot 2024-04-28T12:44:39.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:39 smithi161 ceph-mon[62387]: osdmap e38: 6 total, 6 up, 6 in 2024-04-28T12:44:39.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:39 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-28T12:44:39.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:39 smithi161 ceph-mon[62387]: osdmap e39: 6 total, 6 up, 6 in 2024-04-28T12:44:39.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:39 smithi161 ceph-mon[62387]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-04-28T12:44:40.866 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-28T12:44:40.866 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-28T12:44:40.866 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10+0 records in 2024-04-28T12:44:40.866 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2024-04-28T12:44:40.866 INFO:teuthology.orchestra.run.smithi084.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.036936 s, 284 MB/s 2024-04-28T12:44:40.866 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping successful for: 2024-04-28T12:44:41.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:41 smithi084 ceph-mon[28081]: osdmap e40: 6 total, 6 up, 6 in 2024-04-28T12:44:41.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:41 smithi084 ceph-mon[28081]: Detected new or changed devices on smithi179 2024-04-28T12:44:41.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:41.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:41.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:41.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:41.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:41 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi179 to 8544M 2024-04-28T12:44:41.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:41.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:41.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:41.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:41.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:41 smithi179 ceph-mon[63029]: osdmap e40: 6 total, 6 up, 6 in 2024-04-28T12:44:41.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:41 smithi179 ceph-mon[63029]: Detected new or changed devices on smithi179 2024-04-28T12:44:41.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:41.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:41.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:41.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:41.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:41 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi179 to 8544M 2024-04-28T12:44:41.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:41.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:41.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:41.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:41.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:41 smithi161 ceph-mon[62387]: osdmap e40: 6 total, 6 up, 6 in 2024-04-28T12:44:41.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:41 smithi161 ceph-mon[62387]: Detected new or changed devices on smithi179 2024-04-28T12:44:41.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:41.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:41.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:41.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:44:41.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:41 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi179 to 8544M 2024-04-28T12:44:41.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:44:41.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:44:41.894 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi084:vg_nvme/lv_2 2024-04-28T12:44:42.313 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:44:42.688 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:42 smithi084 ceph-mon[28081]: osdmap e41: 6 total, 6 up, 6 in 2024-04-28T12:44:42.688 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:42 smithi084 ceph-mon[28081]: pgmap v157: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-04-28T12:44:42.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:42 smithi161 ceph-mon[62387]: osdmap e41: 6 total, 6 up, 6 in 2024-04-28T12:44:42.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:42 smithi161 ceph-mon[62387]: pgmap v157: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-04-28T12:44:42.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:42 smithi179 ceph-mon[63029]: osdmap e41: 6 total, 6 up, 6 in 2024-04-28T12:44:42.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:42 smithi179 ceph-mon[63029]: pgmap v157: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-04-28T12:44:43.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:43 smithi084 ceph-mon[28081]: osdmap e42: 6 total, 6 up, 6 in 2024-04-28T12:44:43.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:43 smithi161 ceph-mon[62387]: osdmap e42: 6 total, 6 up, 6 in 2024-04-28T12:44:43.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:43 smithi179 ceph-mon[63029]: osdmap e42: 6 total, 6 up, 6 in 2024-04-28T12:44:44.580 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.584+0000 7f672d400700 1 -- 172.21.15.84:0/2035704224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6728103f10 msgr2=0x7f6728106300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:44.580 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.584+0000 7f672d400700 1 --2- 172.21.15.84:0/2035704224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6728103f10 0x7f6728106300 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f671800b0a0 tx=0x7f671800b3b0 comp rx=0 tx=0).stop 2024-04-28T12:44:44.580 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.584+0000 7f672d400700 1 -- 172.21.15.84:0/2035704224 shutdown_connections 2024-04-28T12:44:44.580 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.584+0000 7f672d400700 1 --2- 172.21.15.84:0/2035704224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6728109170 0x7f672810b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:44.580 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.584+0000 7f672d400700 1 --2- 172.21.15.84:0/2035704224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6728103f10 0x7f6728106300 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:44.580 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.584+0000 7f672d400700 1 --2- 172.21.15.84:0/2035704224 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6728106840 0x7f6728108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:44.580 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.584+0000 7f672d400700 1 -- 172.21.15.84:0/2035704224 >> 172.21.15.84:0/2035704224 conn(0x7f67280fd920 msgr2=0x7f67280ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:44:44.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.584+0000 7f672d400700 1 -- 172.21.15.84:0/2035704224 shutdown_connections 2024-04-28T12:44:44.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.584+0000 7f672d400700 1 -- 172.21.15.84:0/2035704224 wait complete. 2024-04-28T12:44:44.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.585+0000 7f672d400700 1 Processor -- start 2024-04-28T12:44:44.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.585+0000 7f672d400700 1 -- start start 2024-04-28T12:44:44.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.586+0000 7f672d400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6728103f10 0x7f672819b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:44.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.586+0000 7f672d400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6728106840 0x7f672819bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.586+0000 7f672d400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6728109170 0x7f67281a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.586+0000 7f672d400700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f672810ef40 con 0x7f6728103f10 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.586+0000 7f672d400700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f672810edc0 con 0x7f6728109170 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.586+0000 7f672d400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f672810f0c0 con 0x7f6728106840 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.586+0000 7f6726800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6728103f10 0x7f672819b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.586+0000 7f6726800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6728103f10 0x7f672819b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:39778/0 (socket says 172.21.15.84:39778) 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.586+0000 7f6727200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6728109170 0x7f67281a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.586+0000 7f6726800700 1 -- 172.21.15.84:0/3201545039 learned_addr learned my addr 172.21.15.84:0/3201545039 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.586+0000 7f671fe00700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6728106840 0x7f672819bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.587+0000 7f6727200700 1 -- 172.21.15.84:0/3201545039 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6728106840 msgr2=0x7f672819bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.587+0000 7f6727200700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6728106840 0x7f672819bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.587+0000 7f6727200700 1 -- 172.21.15.84:0/3201545039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6728103f10 msgr2=0x7f672819b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:44:44.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.587+0000 7f6727200700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6728103f10 0x7f672819b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:44:44.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.587+0000 7f6727200700 1 -- 172.21.15.84:0/3201545039 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67281a3440 con 0x7f6728109170 2024-04-28T12:44:44.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.587+0000 7f6726800700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6728103f10 0x7f672819b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:44:44.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.587+0000 7f6727200700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6728109170 0x7f67281a2da0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f672000e940 tx=0x7f672000ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:44:44.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.587+0000 7f671f400700 1 -- 172.21.15.84:0/3201545039 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720004220 con 0x7f6728109170 2024-04-28T12:44:44.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.587+0000 7f672d400700 1 -- 172.21.15.84:0/3201545039 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67281a3730 con 0x7f6728109170 2024-04-28T12:44:44.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.587+0000 7f671f400700 1 -- 172.21.15.84:0/3201545039 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67200043c0 con 0x7f6728109170 2024-04-28T12:44:44.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.587+0000 7f671f400700 1 -- 172.21.15.84:0/3201545039 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720010700 con 0x7f6728109170 2024-04-28T12:44:44.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.587+0000 7f672d400700 1 -- 172.21.15.84:0/3201545039 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67281a3c70 con 0x7f6728109170 2024-04-28T12:44:44.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.588+0000 7f672d400700 1 -- 172.21.15.84:0/3201545039 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f672806a110 con 0x7f6728109170 2024-04-28T12:44:44.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.590+0000 7f671f400700 1 -- 172.21.15.84:0/3201545039 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6720004750 con 0x7f6728109170 2024-04-28T12:44:44.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.592+0000 7f671f400700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f670c0a3c40 0x7f670c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:44:44.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.592+0000 7f671f400700 1 -- 172.21.15.84:0/3201545039 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f67200cf330 con 0x7f6728109170 2024-04-28T12:44:44.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.592+0000 7f6726800700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f670c0a3c40 0x7f670c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:44:44.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.593+0000 7f6726800700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f670c0a3c40 0x7f670c0a6100 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6718009be0 tx=0x7f67180060c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:44:44.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.596+0000 7f671f400700 1 -- 172.21.15.84:0/3201545039 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f672009cb30 con 0x7f6728109170 2024-04-28T12:44:44.666 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:44 smithi084 ceph-mon[28081]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-28T12:44:44.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:44 smithi161 ceph-mon[62387]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-28T12:44:44.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:44 smithi179 ceph-mon[63029]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-28T12:44:44.829 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:44:44.833+0000 7f672d400700 1 -- 172.21.15.84:0/3201545039 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6728063860 con 0x7f670c0a3c40 2024-04-28T12:44:45.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:45 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:45.809 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:45 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:45.809 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:45 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:45.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:45 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:45.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:45 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:45.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:45 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:45.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:45 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:44:45.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:45 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:44:45.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:45 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:46.709 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:46 smithi084 ceph-mon[28081]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:46.709 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:46 smithi084 ceph-mon[28081]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T12:44:46.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:46 smithi161 ceph-mon[62387]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:46.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:46 smithi161 ceph-mon[62387]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T12:44:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:46 smithi179 ceph-mon[63029]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:44:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:46 smithi179 ceph-mon[63029]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-28T12:44:48.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:48 smithi179 ceph-mon[63029]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-28T12:44:48.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:48 smithi084 ceph-mon[28081]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-28T12:44:48.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:48 smithi161 ceph-mon[62387]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-28T12:44:49.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:49 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/366414177' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1057554e-494f-4f9e-9823-a9e63c483a5f"}]: dispatch 2024-04-28T12:44:49.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:49 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/366414177' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1057554e-494f-4f9e-9823-a9e63c483a5f"}]': finished 2024-04-28T12:44:49.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:49 smithi179 ceph-mon[63029]: osdmap e43: 7 total, 6 up, 7 in 2024-04-28T12:44:49.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:44:49.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:49 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/366414177' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1057554e-494f-4f9e-9823-a9e63c483a5f"}]: dispatch 2024-04-28T12:44:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:49 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/366414177' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1057554e-494f-4f9e-9823-a9e63c483a5f"}]': finished 2024-04-28T12:44:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:49 smithi084 ceph-mon[28081]: osdmap e43: 7 total, 6 up, 7 in 2024-04-28T12:44:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:44:49.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:49 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/366414177' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1057554e-494f-4f9e-9823-a9e63c483a5f"}]: dispatch 2024-04-28T12:44:49.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:49 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/366414177' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1057554e-494f-4f9e-9823-a9e63c483a5f"}]': finished 2024-04-28T12:44:49.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:49 smithi161 ceph-mon[62387]: osdmap e43: 7 total, 6 up, 7 in 2024-04-28T12:44:49.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:44:50.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:50 smithi179 ceph-mon[63029]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:50.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:50 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1954413792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:50.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:50 smithi084 ceph-mon[28081]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:50.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:50 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1954413792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:50.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:50 smithi161 ceph-mon[62387]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:50.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:50 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1954413792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:44:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:52 smithi179 ceph-mon[63029]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:52.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:52 smithi084 ceph-mon[28081]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:52.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:52 smithi161 ceph-mon[62387]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:54.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:54 smithi179 ceph-mon[63029]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:54.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:54 smithi084 ceph-mon[28081]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:54.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:54 smithi161 ceph-mon[62387]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:56.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:56 smithi084 ceph-mon[28081]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:56.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:56 smithi179 ceph-mon[63029]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:56.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:56 smithi161 ceph-mon[62387]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:58.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:58 smithi179 ceph-mon[63029]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:58.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:58 smithi084 ceph-mon[28081]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:58.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:58 smithi161 ceph-mon[62387]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:44:59.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-28T12:44:59.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:44:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:59.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:59 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-28T12:44:59.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:44:59 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:44:59.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:59 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-28T12:44:59.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:44:59 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:00.894 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:00 smithi161 ceph-mon[62387]: Deploying daemon osd.6 on smithi084 2024-04-28T12:45:00.894 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:00 smithi161 ceph-mon[62387]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:00.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:00 smithi179 ceph-mon[63029]: Deploying daemon osd.6 on smithi084 2024-04-28T12:45:00.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:00 smithi179 ceph-mon[63029]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:00.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:00 smithi084 ceph-mon[28081]: Deploying daemon osd.6 on smithi084 2024-04-28T12:45:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:00 smithi084 ceph-mon[28081]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:02.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:02 smithi179 ceph-mon[63029]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:02.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:02 smithi084 ceph-mon[28081]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:02.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:02 smithi161 ceph-mon[62387]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:03.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:03 smithi084 ceph-mon[28081]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:03.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:03 smithi161 ceph-mon[62387]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:04.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:03 smithi179 ceph-mon[63029]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:06 smithi179 ceph-mon[63029]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:06.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:06 smithi084 ceph-mon[28081]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:06.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:06 smithi161 ceph-mon[62387]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:08.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:08 smithi084 ceph-mon[28081]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:08.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:08.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:08.889 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:08.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:08 smithi179 ceph-mon[63029]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:08.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:08.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:08.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:08 smithi161 ceph-mon[62387]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:10.759 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:10 smithi084 ceph-mon[28081]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:10.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:10 smithi179 ceph-mon[63029]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:10.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:10 smithi161 ceph-mon[62387]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:12.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.434+0000 7f671f400700 1 -- 172.21.15.84:0/3201545039 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6728063860 con 0x7f670c0a3c40 2024-04-28T12:45:12.433 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 6 on host 'smithi084' 2024-04-28T12:45:12.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.438+0000 7f672d400700 1 -- 172.21.15.84:0/3201545039 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f670c0a3c40 msgr2=0x7f670c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:12.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.438+0000 7f672d400700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f670c0a3c40 0x7f670c0a6100 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6718009be0 tx=0x7f67180060c0 comp rx=0 tx=0).stop 2024-04-28T12:45:12.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.438+0000 7f672d400700 1 -- 172.21.15.84:0/3201545039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6728109170 msgr2=0x7f67281a2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:12.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.438+0000 7f672d400700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6728109170 0x7f67281a2da0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f672000e940 tx=0x7f672000ec50 comp rx=0 tx=0).stop 2024-04-28T12:45:12.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.438+0000 7f672d400700 1 -- 172.21.15.84:0/3201545039 shutdown_connections 2024-04-28T12:45:12.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.438+0000 7f672d400700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f670c0a3c40 0x7f670c0a6100 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:12.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.438+0000 7f672d400700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6728109170 0x7f67281a2da0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:12.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.438+0000 7f672d400700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6728103f10 0x7f672819b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:12.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.438+0000 7f672d400700 1 --2- 172.21.15.84:0/3201545039 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6728106840 0x7f672819bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:12.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.438+0000 7f672d400700 1 -- 172.21.15.84:0/3201545039 >> 172.21.15.84:0/3201545039 conn(0x7f67280fd920 msgr2=0x7f6728103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:45:12.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.439+0000 7f672d400700 1 -- 172.21.15.84:0/3201545039 shutdown_connections 2024-04-28T12:45:12.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:45:12.439+0000 7f672d400700 1 -- 172.21.15.84:0/3201545039 wait complete. 2024-04-28T12:45:12.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:12.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:12.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:12 smithi161 ceph-mon[62387]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:12.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:12 smithi161 ceph-mon[62387]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-28T12:45:12.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:12.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:12.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:12.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:12.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:12 smithi084 ceph-mon[28081]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:12.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:12 smithi084 ceph-mon[28081]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-28T12:45:12.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:12.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:12 smithi179 ceph-mon[63029]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:12 smithi179 ceph-mon[63029]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-28T12:45:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:13.426 DEBUG:teuthology.orchestra.run.smithi084:osd.6> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.6.service 2024-04-28T12:45:13.433 INFO:tasks.cephadm:Deploying osd.7 on smithi161 with /dev/vg_nvme/lv_2... 2024-04-28T12:45:13.433 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-28T12:45:13.666 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:13 smithi084 ceph-mon[28081]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-28T12:45:13.666 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:13 smithi084 ceph-mon[28081]: osdmap e44: 7 total, 6 up, 7 in 2024-04-28T12:45:13.666 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:13 smithi084 ceph-mon[28081]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-04-28T12:45:13.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:13.681 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:45:13.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:13 smithi161 ceph-mon[62387]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-28T12:45:13.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:13 smithi161 ceph-mon[62387]: osdmap e44: 7 total, 6 up, 7 in 2024-04-28T12:45:13.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:13 smithi161 ceph-mon[62387]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-04-28T12:45:13.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:13.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:13 smithi179 ceph-mon[63029]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-28T12:45:13.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:13 smithi179 ceph-mon[63029]: osdmap e44: 7 total, 6 up, 7 in 2024-04-28T12:45:13.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:13 smithi179 ceph-mon[63029]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-04-28T12:45:13.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:14.342 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 12:45:14 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6[53405]: 2024-04-28T12:45:14.127+0000 7f75e9800700 -1 osd.6 0 waiting for initial osdmap 2024-04-28T12:45:14.342 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 12:45:14 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6[53405]: 2024-04-28T12:45:14.169+0000 7f75df600700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:45:14.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:14 smithi084 ceph-mon[28081]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-04-28T12:45:14.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:14 smithi084 ceph-mon[28081]: osdmap e45: 7 total, 6 up, 7 in 2024-04-28T12:45:14.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:14.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:14.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:14 smithi084 ceph-mon[28081]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:14.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:14 smithi084 ceph-mon[28081]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' 2024-04-28T12:45:14.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:14 smithi179 ceph-mon[63029]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-04-28T12:45:14.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:14 smithi179 ceph-mon[63029]: osdmap e45: 7 total, 6 up, 7 in 2024-04-28T12:45:14.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:14.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:14.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:14 smithi179 ceph-mon[63029]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:14.816 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:14 smithi179 ceph-mon[63029]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' 2024-04-28T12:45:14.840 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:14 smithi161 ceph-mon[62387]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-04-28T12:45:14.840 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:14 smithi161 ceph-mon[62387]: osdmap e45: 7 total, 6 up, 7 in 2024-04-28T12:45:14.841 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:14.841 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:14.841 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:14 smithi161 ceph-mon[62387]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-04-28T12:45:14.841 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:14 smithi161 ceph-mon[62387]: from='osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785]' entity='osd.6' 2024-04-28T12:45:15.710 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-28T12:45:15.710 INFO:teuthology.orchestra.run.smithi161.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-28T12:45:15.710 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10+0 records in 2024-04-28T12:45:15.710 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2024-04-28T12:45:15.710 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112819 s, 929 MB/s 2024-04-28T12:45:15.710 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping successful for: 2024-04-28T12:45:15.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:15 smithi084 ceph-mon[28081]: purged_snaps scrub starts 2024-04-28T12:45:15.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:15 smithi084 ceph-mon[28081]: purged_snaps scrub ok 2024-04-28T12:45:15.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:15 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:15.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:15 smithi084 ceph-mon[28081]: osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785] boot 2024-04-28T12:45:15.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:15 smithi084 ceph-mon[28081]: osdmap e46: 7 total, 7 up, 7 in 2024-04-28T12:45:15.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:15 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:15.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:15 smithi179 ceph-mon[63029]: purged_snaps scrub starts 2024-04-28T12:45:15.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:15 smithi179 ceph-mon[63029]: purged_snaps scrub ok 2024-04-28T12:45:15.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:15 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:15.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:15 smithi179 ceph-mon[63029]: osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785] boot 2024-04-28T12:45:15.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:15 smithi179 ceph-mon[63029]: osdmap e46: 7 total, 7 up, 7 in 2024-04-28T12:45:15.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:15 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:15.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:15 smithi161 ceph-mon[62387]: purged_snaps scrub starts 2024-04-28T12:45:15.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:15 smithi161 ceph-mon[62387]: purged_snaps scrub ok 2024-04-28T12:45:15.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:15 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:15.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:15 smithi161 ceph-mon[62387]: osd.6 [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785] boot 2024-04-28T12:45:15.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:15 smithi161 ceph-mon[62387]: osdmap e46: 7 total, 7 up, 7 in 2024-04-28T12:45:15.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:15 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-28T12:45:16.377 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi161:vg_nvme/lv_2 2024-04-28T12:45:16.621 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:45:16.657 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:16 smithi161 ceph-mon[62387]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:16.657 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:16 smithi161 ceph-mon[62387]: osdmap e47: 7 total, 7 up, 7 in 2024-04-28T12:45:16.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:16 smithi179 ceph-mon[63029]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:16.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:16 smithi179 ceph-mon[63029]: osdmap e47: 7 total, 7 up, 7 in 2024-04-28T12:45:16.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:16 smithi084 ceph-mon[28081]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:16.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:16 smithi084 ceph-mon[28081]: osdmap e47: 7 total, 7 up, 7 in 2024-04-28T12:45:18.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.411+0000 7fdbb1304700 1 -- 172.21.15.161:0/904668675 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbac104690 msgr2=0x7fdbac104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:18.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.411+0000 7fdbb1304700 1 --2- 172.21.15.161:0/904668675 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbac104690 0x7fdbac104af0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdb940099c0 tx=0x7fdb94009cd0 comp rx=0 tx=0).stop 2024-04-28T12:45:18.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.411+0000 7fdbb1304700 1 -- 172.21.15.161:0/904668675 shutdown_connections 2024-04-28T12:45:18.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.411+0000 7fdbb1304700 1 --2- 172.21.15.161:0/904668675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdbac105030 0x7fdbac10b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:18.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.411+0000 7fdbb1304700 1 --2- 172.21.15.161:0/904668675 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdbac103490 0x7fdbac1038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:18.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.411+0000 7fdbb1304700 1 --2- 172.21.15.161:0/904668675 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbac104690 0x7fdbac104af0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:18.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.411+0000 7fdbb1304700 1 -- 172.21.15.161:0/904668675 >> 172.21.15.161:0/904668675 conn(0x7fdbac0fea30 msgr2=0x7fdbac100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:45:18.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.411+0000 7fdbb1304700 1 -- 172.21.15.161:0/904668675 shutdown_connections 2024-04-28T12:45:18.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.411+0000 7fdbb1304700 1 -- 172.21.15.161:0/904668675 wait complete. 2024-04-28T12:45:18.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.411+0000 7fdbb1304700 1 Processor -- start 2024-04-28T12:45:18.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbb1304700 1 -- start start 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbb1304700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbac103490 0x7fdbac199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbb1304700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdbac104690 0x7fdbac199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbb1304700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdbac105030 0x7fdbac1a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbb1304700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac10efd0 con 0x7fdbac104690 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbb1304700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac10ee50 con 0x7fdbac103490 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbb1304700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac10f150 con 0x7fdbac105030 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbaaffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbac103490 0x7fdbac199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbaaffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbac103490 0x7fdbac199610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37766/0 (socket says 172.21.15.161:37766) 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbaaffd700 1 -- 172.21.15.161:0/829600504 learned_addr learned my addr 172.21.15.161:0/829600504 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbab7fe700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdbac105030 0x7fdbac1a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbaaffd700 1 -- 172.21.15.161:0/829600504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdbac105030 msgr2=0x7fdbac1a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbaaffd700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdbac105030 0x7fdbac1a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:18.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbaaffd700 1 -- 172.21.15.161:0/829600504 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdbac104690 msgr2=0x7fdbac199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:18.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbaaffd700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdbac104690 0x7fdbac199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:18.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbaaffd700 1 -- 172.21.15.161:0/829600504 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbac1a1270 con 0x7fdbac103490 2024-04-28T12:45:18.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.412+0000 7fdbaaffd700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbac103490 0x7fdbac199610 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00b500 tx=0x7fdb9c00b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:45:18.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.413+0000 7fdba8ff9700 1 -- 172.21.15.161:0/829600504 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c015020 con 0x7fdbac103490 2024-04-28T12:45:18.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.413+0000 7fdbb1304700 1 -- 172.21.15.161:0/829600504 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbac1a1560 con 0x7fdbac103490 2024-04-28T12:45:18.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.413+0000 7fdba8ff9700 1 -- 172.21.15.161:0/829600504 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb9c011ce0 con 0x7fdbac103490 2024-04-28T12:45:18.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.413+0000 7fdba8ff9700 1 -- 172.21.15.161:0/829600504 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c010d50 con 0x7fdbac103490 2024-04-28T12:45:18.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.413+0000 7fdbb1304700 1 -- 172.21.15.161:0/829600504 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbac1a1af0 con 0x7fdbac103490 2024-04-28T12:45:18.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.414+0000 7fdbb1304700 1 -- 172.21.15.161:0/829600504 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbac06a110 con 0x7fdbac103490 2024-04-28T12:45:18.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.415+0000 7fdba8ff9700 1 -- 172.21.15.161:0/829600504 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdb9c018020 con 0x7fdbac103490 2024-04-28T12:45:18.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.415+0000 7fdba8ff9700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fdb980a3cc0 0x7fdb980a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:18.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.415+0000 7fdba8ff9700 1 -- 172.21.15.161:0/829600504 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fdb9c0ced80 con 0x7fdbac103490 2024-04-28T12:45:18.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.416+0000 7fdba27fc700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fdb980a3cc0 0x7fdb980a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:18.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.417+0000 7fdba27fc700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fdb980a3cc0 0x7fdb980a6180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdb94003790 tx=0x7fdb94009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:45:18.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.420+0000 7fdba8ff9700 1 -- 172.21.15.161:0/829600504 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdb9c01f020 con 0x7fdbac103490 2024-04-28T12:45:18.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:18.629+0000 7fdbb1304700 1 -- 172.21.15.161:0/829600504 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fdbac063860 con 0x7fdb980a3cc0 2024-04-28T12:45:18.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:18 smithi179 ceph-mon[63029]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:18.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:18 smithi084 ceph-mon[28081]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:18.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:18 smithi161 ceph-mon[62387]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:19.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:45:19.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:45:19.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:19.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:19.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:19.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:19 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:45:19.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:45:19.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:19.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:19.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:19.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:19.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:19.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:19.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:19.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:19.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:19.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:19 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:45:19.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:45:19.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:19.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:19.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:19.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:19.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:19.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:19.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:19.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:19.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:19.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:19.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:19 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:20.617 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:20 smithi161 ceph-mon[62387]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:45:20.618 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:20 smithi161 ceph-mon[62387]: Detected new or changed devices on smithi084 2024-04-28T12:45:20.618 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:20 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi084 to 5696M 2024-04-28T12:45:20.618 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:20 smithi161 ceph-mon[62387]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:20 smithi179 ceph-mon[63029]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:45:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:20 smithi179 ceph-mon[63029]: Detected new or changed devices on smithi084 2024-04-28T12:45:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:20 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi084 to 5696M 2024-04-28T12:45:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:20 smithi179 ceph-mon[63029]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:20 smithi084 ceph-mon[28081]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:45:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:20 smithi084 ceph-mon[28081]: Detected new or changed devices on smithi084 2024-04-28T12:45:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:20 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi084 to 5696M 2024-04-28T12:45:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:20 smithi084 ceph-mon[28081]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:21.798 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:21 smithi161 ceph-mon[62387]: from='client.? 172.21.15.161:0/4116942289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7108aa59-cf9a-4903-89fa-d0c456f0ed1d"}]: dispatch 2024-04-28T12:45:21.798 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:21 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7108aa59-cf9a-4903-89fa-d0c456f0ed1d"}]: dispatch 2024-04-28T12:45:21.798 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:21 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7108aa59-cf9a-4903-89fa-d0c456f0ed1d"}]': finished 2024-04-28T12:45:21.798 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:21 smithi161 ceph-mon[62387]: osdmap e48: 8 total, 7 up, 8 in 2024-04-28T12:45:21.798 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:21 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:21.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:21 smithi179 ceph-mon[63029]: from='client.? 172.21.15.161:0/4116942289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7108aa59-cf9a-4903-89fa-d0c456f0ed1d"}]: dispatch 2024-04-28T12:45:21.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:21 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7108aa59-cf9a-4903-89fa-d0c456f0ed1d"}]: dispatch 2024-04-28T12:45:21.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:21 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7108aa59-cf9a-4903-89fa-d0c456f0ed1d"}]': finished 2024-04-28T12:45:21.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:21 smithi179 ceph-mon[63029]: osdmap e48: 8 total, 7 up, 8 in 2024-04-28T12:45:21.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:21 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:21.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:21 smithi084 ceph-mon[28081]: from='client.? 172.21.15.161:0/4116942289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7108aa59-cf9a-4903-89fa-d0c456f0ed1d"}]: dispatch 2024-04-28T12:45:21.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:21 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7108aa59-cf9a-4903-89fa-d0c456f0ed1d"}]: dispatch 2024-04-28T12:45:21.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:21 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7108aa59-cf9a-4903-89fa-d0c456f0ed1d"}]': finished 2024-04-28T12:45:21.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:21 smithi084 ceph-mon[28081]: osdmap e48: 8 total, 7 up, 8 in 2024-04-28T12:45:21.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:21 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:22 smithi179 ceph-mon[63029]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:22 smithi179 ceph-mon[63029]: from='client.? 172.21.15.161:0/2246302217' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:45:22.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:22 smithi084 ceph-mon[28081]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:22.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:22 smithi084 ceph-mon[28081]: from='client.? 172.21.15.161:0/2246302217' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:45:22.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:22 smithi161 ceph-mon[62387]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:22.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:22 smithi161 ceph-mon[62387]: from='client.? 172.21.15.161:0/2246302217' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:45:24.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:24 smithi179 ceph-mon[63029]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:24.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:24 smithi084 ceph-mon[28081]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:24.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:24 smithi161 ceph-mon[62387]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:26.722 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:26 smithi161 ceph-mon[62387]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:26.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:26 smithi179 ceph-mon[63029]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:26.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:26 smithi084 ceph-mon[28081]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:28.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:28 smithi161 ceph-mon[62387]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:28.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:28 smithi179 ceph-mon[63029]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:28.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:28 smithi084 ceph-mon[28081]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:29.786 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-28T12:45:29.787 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:29.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-28T12:45:29.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:29.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-28T12:45:29.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:30.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:30 smithi161 ceph-mon[62387]: Deploying daemon osd.7 on smithi161 2024-04-28T12:45:30.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:30 smithi161 ceph-mon[62387]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:30.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:30 smithi179 ceph-mon[63029]: Deploying daemon osd.7 on smithi161 2024-04-28T12:45:30.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:30 smithi179 ceph-mon[63029]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:30.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:30 smithi084 ceph-mon[28081]: Deploying daemon osd.7 on smithi161 2024-04-28T12:45:30.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:30 smithi084 ceph-mon[28081]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:32.690 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:32 smithi161 ceph-mon[62387]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:32 smithi179 ceph-mon[63029]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:32.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:32 smithi084 ceph-mon[28081]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:34 smithi179 ceph-mon[63029]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:34.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:34 smithi084 ceph-mon[28081]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:34.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:34.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:34.951 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:34 smithi161 ceph-mon[62387]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:34.951 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:34.951 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:35.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:35 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:35.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:35 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:35.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:35 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:36.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:36 smithi179 ceph-mon[63029]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:36.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:36 smithi084 ceph-mon[28081]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:36.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:36 smithi161 ceph-mon[62387]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:38.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.374+0000 7fdba8ff9700 1 -- 172.21.15.161:0/829600504 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdbac063860 con 0x7fdb980a3cc0 2024-04-28T12:45:38.371 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 7 on host 'smithi161' 2024-04-28T12:45:38.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.376+0000 7fdbb1304700 1 -- 172.21.15.161:0/829600504 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fdb980a3cc0 msgr2=0x7fdb980a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:38.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.376+0000 7fdbb1304700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fdb980a3cc0 0x7fdb980a6180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdb94003790 tx=0x7fdb94009f10 comp rx=0 tx=0).stop 2024-04-28T12:45:38.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.376+0000 7fdbb1304700 1 -- 172.21.15.161:0/829600504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbac103490 msgr2=0x7fdbac199610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:38.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.376+0000 7fdbb1304700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbac103490 0x7fdbac199610 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00b500 tx=0x7fdb9c00b810 comp rx=0 tx=0).stop 2024-04-28T12:45:38.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.376+0000 7fdbb1304700 1 -- 172.21.15.161:0/829600504 shutdown_connections 2024-04-28T12:45:38.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.376+0000 7fdbb1304700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fdb980a3cc0 0x7fdb980a6180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:38.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.376+0000 7fdbb1304700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdbac103490 0x7fdbac199610 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:38.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.376+0000 7fdbb1304700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdbac105030 0x7fdbac1a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:38.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.376+0000 7fdbb1304700 1 --2- 172.21.15.161:0/829600504 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdbac104690 0x7fdbac199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:38.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.376+0000 7fdbb1304700 1 -- 172.21.15.161:0/829600504 >> 172.21.15.161:0/829600504 conn(0x7fdbac0fea30 msgr2=0x7fdbac107990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:45:38.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.376+0000 7fdbb1304700 1 -- 172.21.15.161:0/829600504 shutdown_connections 2024-04-28T12:45:38.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:45:38.376+0000 7fdbb1304700 1 -- 172.21.15.161:0/829600504 wait complete. 2024-04-28T12:45:38.698 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.698 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:38 smithi161 ceph-mon[62387]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:38.698 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.698 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:38.698 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:38.698 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.698 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.698 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.698 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:38.698 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:38 smithi161 ceph-mon[62387]: from='osd.7 [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T12:45:38.699 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:38 smithi161 ceph-mon[62387]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T12:45:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:38 smithi179 ceph-mon[63029]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:38 smithi179 ceph-mon[63029]: from='osd.7 [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T12:45:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:38 smithi179 ceph-mon[63029]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T12:45:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:38 smithi084 ceph-mon[28081]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:45:38.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:38 smithi084 ceph-mon[28081]: from='osd.7 [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T12:45:38.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:38 smithi084 ceph-mon[28081]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-28T12:45:39.058 DEBUG:teuthology.orchestra.run.smithi161:osd.7> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.7.service 2024-04-28T12:45:39.060 INFO:tasks.cephadm:Deploying osd.8 on smithi179 with /dev/vg_nvme/lv_2... 2024-04-28T12:45:39.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-28T12:45:39.307 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:45:40.478 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:40 smithi179 ceph-mon[63029]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-28T12:45:40.478 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:40 smithi179 ceph-mon[63029]: from='osd.7 [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:45:40.478 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:40 smithi179 ceph-mon[63029]: osdmap e49: 8 total, 7 up, 8 in 2024-04-28T12:45:40.478 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:40 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:40.478 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:40 smithi179 ceph-mon[63029]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:45:40.478 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:40 smithi179 ceph-mon[63029]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:40.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:40 smithi161 ceph-mon[62387]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-28T12:45:40.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:40 smithi161 ceph-mon[62387]: from='osd.7 [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:45:40.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:40 smithi161 ceph-mon[62387]: osdmap e49: 8 total, 7 up, 8 in 2024-04-28T12:45:40.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:40 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:40.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:40 smithi161 ceph-mon[62387]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:45:40.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:40 smithi161 ceph-mon[62387]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:40.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:40 smithi084 ceph-mon[28081]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-28T12:45:40.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:40 smithi084 ceph-mon[28081]: from='osd.7 [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:45:40.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:40 smithi084 ceph-mon[28081]: osdmap e49: 8 total, 7 up, 8 in 2024-04-28T12:45:40.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:40 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:40.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:40 smithi084 ceph-mon[28081]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:45:40.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:40 smithi084 ceph-mon[28081]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-04-28T12:45:41.243 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 12:45:40 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7[80463]: 2024-04-28T12:45:40.794+0000 7f20b45bc700 -1 osd.7 0 waiting for initial osdmap 2024-04-28T12:45:41.243 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 12:45:40 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7[80463]: 2024-04-28T12:45:40.804+0000 7f20ac3a7700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:45:41.404 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-28T12:45:41.404 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-28T12:45:41.404 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-04-28T12:45:41.404 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-04-28T12:45:41.404 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112939 s, 928 MB/s 2024-04-28T12:45:41.405 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-04-28T12:45:41.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:41 smithi179 ceph-mon[63029]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-04-28T12:45:41.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:41 smithi179 ceph-mon[63029]: osdmap e50: 8 total, 7 up, 8 in 2024-04-28T12:45:41.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:41.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:41 smithi179 ceph-mon[63029]: from='osd.7 ' entity='osd.7' 2024-04-28T12:45:41.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:41 smithi179 ceph-mon[63029]: osd.7 [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087] boot 2024-04-28T12:45:41.431 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:41 smithi179 ceph-mon[63029]: osdmap e51: 8 total, 8 up, 8 in 2024-04-28T12:45:41.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:41 smithi084 ceph-mon[28081]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-04-28T12:45:41.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:41 smithi084 ceph-mon[28081]: osdmap e50: 8 total, 7 up, 8 in 2024-04-28T12:45:41.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:41.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:41 smithi084 ceph-mon[28081]: from='osd.7 ' entity='osd.7' 2024-04-28T12:45:41.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:41 smithi084 ceph-mon[28081]: osd.7 [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087] boot 2024-04-28T12:45:41.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:41 smithi084 ceph-mon[28081]: osdmap e51: 8 total, 8 up, 8 in 2024-04-28T12:45:41.705 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:41 smithi161 ceph-mon[62387]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-04-28T12:45:41.705 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:41 smithi161 ceph-mon[62387]: osdmap e50: 8 total, 7 up, 8 in 2024-04-28T12:45:41.705 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:41.705 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:41 smithi161 ceph-mon[62387]: from='osd.7 ' entity='osd.7' 2024-04-28T12:45:41.705 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:41 smithi161 ceph-mon[62387]: osd.7 [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087] boot 2024-04-28T12:45:41.705 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:41 smithi161 ceph-mon[62387]: osdmap e51: 8 total, 8 up, 8 in 2024-04-28T12:45:42.157 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi179:vg_nvme/lv_2 2024-04-28T12:45:42.391 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: purged_snaps scrub starts 2024-04-28T12:45:42.391 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: purged_snaps scrub ok 2024-04-28T12:45:42.391 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:42.391 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:42.391 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:42.391 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:42.391 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:42.391 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:42.391 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:42.391 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:42.391 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:42.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:42.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:42.392 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:42 smithi179 ceph-mon[63029]: osdmap e52: 8 total, 8 up, 8 in 2024-04-28T12:45:42.401 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: purged_snaps scrub starts 2024-04-28T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: purged_snaps scrub ok 2024-04-28T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:42.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:42.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:42 smithi084 ceph-mon[28081]: osdmap e52: 8 total, 8 up, 8 in 2024-04-28T12:45:42.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: purged_snaps scrub starts 2024-04-28T12:45:42.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: purged_snaps scrub ok 2024-04-28T12:45:42.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-28T12:45:42.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:42.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:42.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:42.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:42.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:42.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:45:42.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:42.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:42.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:45:42.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:45:42.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:42 smithi161 ceph-mon[62387]: osdmap e52: 8 total, 8 up, 8 in 2024-04-28T12:45:43.534 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:43 smithi179 ceph-mon[63029]: Detected new or changed devices on smithi161 2024-04-28T12:45:43.534 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:43 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi161 to 5696M 2024-04-28T12:45:43.535 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:43 smithi179 ceph-mon[63029]: osdmap e53: 8 total, 8 up, 8 in 2024-04-28T12:45:43.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:43 smithi084 ceph-mon[28081]: Detected new or changed devices on smithi161 2024-04-28T12:45:43.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:43 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi161 to 5696M 2024-04-28T12:45:43.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:43 smithi084 ceph-mon[28081]: osdmap e53: 8 total, 8 up, 8 in 2024-04-28T12:45:43.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:43 smithi161 ceph-mon[62387]: Detected new or changed devices on smithi161 2024-04-28T12:45:43.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:43 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi161 to 5696M 2024-04-28T12:45:43.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:43 smithi161 ceph-mon[62387]: osdmap e53: 8 total, 8 up, 8 in 2024-04-28T12:45:44.063 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.064+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/2767332290 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7a4103f20 msgr2=0x7fa7a410a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:44.063 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.064+0000 7fa7a9f0e700 1 --2- 172.21.15.179:0/2767332290 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7a4103f20 0x7fa7a410a7d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa79c009990 tx=0x7fa79c009ca0 comp rx=0 tx=0).stop 2024-04-28T12:45:44.063 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.064+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/2767332290 shutdown_connections 2024-04-28T12:45:44.063 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.064+0000 7fa7a9f0e700 1 --2- 172.21.15.179:0/2767332290 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7a4103f20 0x7fa7a410a7d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:44.063 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.064+0000 7fa7a9f0e700 1 --2- 172.21.15.179:0/2767332290 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7a4102380 0x7fa7a41027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:44.063 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.064+0000 7fa7a9f0e700 1 --2- 172.21.15.179:0/2767332290 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7a4103580 0x7fa7a41039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:44.063 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.064+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/2767332290 >> 172.21.15.179:0/2767332290 conn(0x7fa7a40fd940 msgr2=0x7fa7a40ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:45:44.063 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.064+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/2767332290 shutdown_connections 2024-04-28T12:45:44.063 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.064+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/2767332290 wait complete. 2024-04-28T12:45:44.063 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.064+0000 7fa7a9f0e700 1 Processor -- start 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.064+0000 7fa7a9f0e700 1 -- start start 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a9f0e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7a4102380 0x7fa7a4198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a9f0e700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7a4103580 0x7fa7a4198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a9f0e700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7a4103f20 0x7fa7a419fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a9f0e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a410de90 con 0x7fa7a4102380 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a9f0e700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a410dd10 con 0x7fa7a4103580 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a9f0e700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a410e010 con 0x7fa7a4103f20 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a3fff700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7a4103f20 0x7fa7a419fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a3fff700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7a4103f20 0x7fa7a419fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:50750/0 (socket says 172.21.15.179:50750) 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a3fff700 1 -- 172.21.15.179:0/1205439596 learned_addr learned my addr 172.21.15.179:0/1205439596 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a2ffd700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7a4103580 0x7fa7a4198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a3fff700 1 -- 172.21.15.179:0/1205439596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7a4103580 msgr2=0x7fa7a4198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:44.064 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a3fff700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7a4103580 0x7fa7a4198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:44.065 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a3fff700 1 -- 172.21.15.179:0/1205439596 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7a4102380 msgr2=0x7fa7a4198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:45:44.065 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a3fff700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7a4102380 0x7fa7a4198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:45:44.065 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a3fff700 1 -- 172.21.15.179:0/1205439596 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a41a00a0 con 0x7fa7a4103f20 2024-04-28T12:45:44.065 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a2ffd700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7a4103580 0x7fa7a4198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:45:44.065 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a3fff700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7a4103f20 0x7fa7a419fa00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa79c005910 tx=0x7fa79c009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:45:44.065 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a0ff9700 1 -- 172.21.15.179:0/1205439596 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c027070 con 0x7fa7a4103f20 2024-04-28T12:45:44.065 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a0ff9700 1 -- 172.21.15.179:0/1205439596 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa79c022470 con 0x7fa7a4103f20 2024-04-28T12:45:44.065 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/1205439596 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a41a0330 con 0x7fa7a4103f20 2024-04-28T12:45:44.065 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a0ff9700 1 -- 172.21.15.179:0/1205439596 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c00f6c0 con 0x7fa7a4103f20 2024-04-28T12:45:44.065 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.065+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/1205439596 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a41a0780 con 0x7fa7a4103f20 2024-04-28T12:45:44.065 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.066+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/1205439596 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a406a110 con 0x7fa7a4103f20 2024-04-28T12:45:44.066 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.067+0000 7fa7a0ff9700 1 -- 172.21.15.179:0/1205439596 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa79c02a070 con 0x7fa7a4103f20 2024-04-28T12:45:44.066 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.068+0000 7fa7a0ff9700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa78c0a3c40 0x7fa78c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:45:44.066 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.068+0000 7fa7a0ff9700 1 -- 172.21.15.179:0/1205439596 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7fa79c0d0a50 con 0x7fa7a4103f20 2024-04-28T12:45:44.066 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.069+0000 7fa7a37fe700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa78c0a3c40 0x7fa78c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:45:44.066 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.069+0000 7fa7a37fe700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa78c0a3c40 0x7fa78c0a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7a41033e0 tx=0x7fa794009250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:45:44.068 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.072+0000 7fa7a0ff9700 1 -- 172.21.15.179:0/1205439596 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa79c013070 con 0x7fa7a4103f20 2024-04-28T12:45:44.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:45:44.281+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/1205439596 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa7a4063860 con 0x7fa78c0a3c40 2024-04-28T12:45:44.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:44 smithi179 ceph-mon[63029]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:44.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:44 smithi179 ceph-mon[63029]: osdmap e54: 8 total, 8 up, 8 in 2024-04-28T12:45:44.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:45:44.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:45:44.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:44.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:44 smithi084 ceph-mon[28081]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:44.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:44 smithi084 ceph-mon[28081]: osdmap e54: 8 total, 8 up, 8 in 2024-04-28T12:45:44.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:45:44.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:45:44.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:44.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:44 smithi161 ceph-mon[62387]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:44.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:44 smithi161 ceph-mon[62387]: osdmap e54: 8 total, 8 up, 8 in 2024-04-28T12:45:44.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:45:44.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:45:44.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:45.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:45 smithi179 ceph-mon[63029]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:45:45.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:45 smithi179 ceph-mon[63029]: osdmap e55: 8 total, 8 up, 8 in 2024-04-28T12:45:45.666 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:45 smithi084 ceph-mon[28081]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:45:45.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:45 smithi084 ceph-mon[28081]: osdmap e55: 8 total, 8 up, 8 in 2024-04-28T12:45:45.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:45 smithi161 ceph-mon[62387]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:45:45.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:45 smithi161 ceph-mon[62387]: osdmap e55: 8 total, 8 up, 8 in 2024-04-28T12:45:46.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:46 smithi179 ceph-mon[63029]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:46.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:46 smithi084 ceph-mon[28081]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:46.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:46 smithi161 ceph-mon[62387]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:47.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:47 smithi084 ceph-mon[28081]: from='client.? 172.21.15.179:0/4114355673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b005d60-667b-4061-80e9-af319ce052df"}]: dispatch 2024-04-28T12:45:47.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:47 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b005d60-667b-4061-80e9-af319ce052df"}]: dispatch 2024-04-28T12:45:47.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:47 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b005d60-667b-4061-80e9-af319ce052df"}]': finished 2024-04-28T12:45:47.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:47 smithi084 ceph-mon[28081]: osdmap e56: 9 total, 8 up, 9 in 2024-04-28T12:45:47.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:45:47.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:47 smithi161 ceph-mon[62387]: from='client.? 172.21.15.179:0/4114355673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b005d60-667b-4061-80e9-af319ce052df"}]: dispatch 2024-04-28T12:45:47.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:47 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b005d60-667b-4061-80e9-af319ce052df"}]: dispatch 2024-04-28T12:45:47.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:47 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b005d60-667b-4061-80e9-af319ce052df"}]': finished 2024-04-28T12:45:47.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:47 smithi161 ceph-mon[62387]: osdmap e56: 9 total, 8 up, 9 in 2024-04-28T12:45:47.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:45:47.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:47 smithi179 ceph-mon[63029]: from='client.? 172.21.15.179:0/4114355673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b005d60-667b-4061-80e9-af319ce052df"}]: dispatch 2024-04-28T12:45:47.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:47 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b005d60-667b-4061-80e9-af319ce052df"}]: dispatch 2024-04-28T12:45:47.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:47 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b005d60-667b-4061-80e9-af319ce052df"}]': finished 2024-04-28T12:45:47.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:47 smithi179 ceph-mon[63029]: osdmap e56: 9 total, 8 up, 9 in 2024-04-28T12:45:47.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:45:48.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:48 smithi161 ceph-mon[62387]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:48.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:48 smithi161 ceph-mon[62387]: from='client.? 172.21.15.179:0/399972770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:45:48.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:48 smithi179 ceph-mon[63029]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:48.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:48 smithi179 ceph-mon[63029]: from='client.? 172.21.15.179:0/399972770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:45:48.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:48 smithi084 ceph-mon[28081]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:48.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:48 smithi084 ceph-mon[28081]: from='client.? 172.21.15.179:0/399972770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:45:50.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:50 smithi179 ceph-mon[63029]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:50.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:50 smithi084 ceph-mon[28081]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:50.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:50 smithi161 ceph-mon[62387]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:52.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:52 smithi179 ceph-mon[63029]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:52.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:52 smithi084 ceph-mon[28081]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:52.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:52 smithi161 ceph-mon[62387]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:54.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:54 smithi179 ceph-mon[63029]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:54.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:54 smithi084 ceph-mon[28081]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:54.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:54 smithi161 ceph-mon[62387]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:55.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-28T12:45:55.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:55.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:55 smithi179 ceph-mon[63029]: Deploying daemon osd.8 on smithi179 2024-04-28T12:45:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-28T12:45:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:55 smithi084 ceph-mon[28081]: Deploying daemon osd.8 on smithi179 2024-04-28T12:45:55.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-28T12:45:55.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:45:55.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:55 smithi161 ceph-mon[62387]: Deploying daemon osd.8 on smithi179 2024-04-28T12:45:56.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:56 smithi179 ceph-mon[63029]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:56.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:56 smithi084 ceph-mon[28081]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:56.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:56 smithi161 ceph-mon[62387]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:58.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:45:58 smithi179 ceph-mon[63029]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:58.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:45:58 smithi084 ceph-mon[28081]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:45:58.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:45:58 smithi161 ceph-mon[62387]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:00 smithi179 ceph-mon[63029]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:00 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:00 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:00 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:00.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:00 smithi084 ceph-mon[28081]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:00 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:00 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:00 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:00.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:00 smithi161 ceph-mon[62387]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:00.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:00 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:00.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:00 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:00.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:00 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:01.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:01 smithi161 ceph-mon[62387]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:02.112 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:01 smithi179 ceph-mon[63029]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:02.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:01 smithi084 ceph-mon[28081]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:03.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.724+0000 7fa7a0ff9700 1 -- 172.21.15.179:0/1205439596 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa7a4063860 con 0x7fa78c0a3c40 2024-04-28T12:46:03.720 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 8 on host 'smithi179' 2024-04-28T12:46:03.722 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.726+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/1205439596 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa78c0a3c40 msgr2=0x7fa78c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:03.722 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.726+0000 7fa7a9f0e700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa78c0a3c40 0x7fa78c0a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7a41033e0 tx=0x7fa794009250 comp rx=0 tx=0).stop 2024-04-28T12:46:03.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.726+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/1205439596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7a4103f20 msgr2=0x7fa7a419fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:03.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.726+0000 7fa7a9f0e700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7a4103f20 0x7fa7a419fa00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa79c005910 tx=0x7fa79c009ee0 comp rx=0 tx=0).stop 2024-04-28T12:46:03.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.726+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/1205439596 shutdown_connections 2024-04-28T12:46:03.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.726+0000 7fa7a9f0e700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa7a4103f20 0x7fa7a419fa00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:03.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.726+0000 7fa7a9f0e700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7a4102380 0x7fa7a4198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:03.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.726+0000 7fa7a9f0e700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa78c0a3c40 0x7fa78c0a6100 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:03.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.726+0000 7fa7a9f0e700 1 --2- 172.21.15.179:0/1205439596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7a4103580 0x7fa7a4198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:03.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.726+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/1205439596 >> 172.21.15.179:0/1205439596 conn(0x7fa7a40fd940 msgr2=0x7fa7a4106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:46:03.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.727+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/1205439596 shutdown_connections 2024-04-28T12:46:03.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:46:03.727+0000 7fa7a9f0e700 1 -- 172.21.15.179:0/1205439596 wait complete. 2024-04-28T12:46:04.833 DEBUG:teuthology.orchestra.run.smithi179:osd.8> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.8.service 2024-04-28T12:46:04.835 INFO:tasks.cephadm:Deploying osd.9 on smithi084 with /dev/vg_nvme/lv_1... 2024-04-28T12:46:04.836 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-28T12:46:04.857 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:04 smithi179 ceph-mon[63029]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:04.857 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:04 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:04.857 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:04 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:04.857 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:04 smithi179 ceph-mon[63029]: from='osd.8 [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T12:46:04.857 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:04 smithi179 ceph-mon[63029]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T12:46:04.915 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:04 smithi084 ceph-mon[28081]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:04.915 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:04 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:04.915 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:04 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:04.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:04 smithi084 ceph-mon[28081]: from='osd.8 [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T12:46:04.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:04 smithi084 ceph-mon[28081]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T12:46:04.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:04 smithi161 ceph-mon[62387]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:04.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:04 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:04.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:04 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:04.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:04 smithi161 ceph-mon[62387]: from='osd.8 [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T12:46:04.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:04 smithi161 ceph-mon[62387]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-28T12:46:05.289 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:46:05.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:05 smithi179 ceph-mon[63029]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-28T12:46:05.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:05 smithi179 ceph-mon[63029]: from='osd.8 [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:46:05.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:05 smithi179 ceph-mon[63029]: osdmap e57: 9 total, 8 up, 9 in 2024-04-28T12:46:05.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:05 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:05.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:05 smithi179 ceph-mon[63029]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:46:05.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:05 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:05.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:05 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:05.840 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:05 smithi179 ceph-mon[63029]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:05.841 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 12:46:05 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8[81674]: 2024-04-28T12:46:05.709+0000 7fcc96b4a700 -1 osd.8 0 waiting for initial osdmap 2024-04-28T12:46:05.841 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 12:46:05 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8[81674]: 2024-04-28T12:46:05.718+0000 7fcc9293d700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:46:05.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:05 smithi161 ceph-mon[62387]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-28T12:46:05.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:05 smithi161 ceph-mon[62387]: from='osd.8 [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:46:05.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:05 smithi161 ceph-mon[62387]: osdmap e57: 9 total, 8 up, 9 in 2024-04-28T12:46:05.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:05 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:05.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:05 smithi161 ceph-mon[62387]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:46:05.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:05 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:05.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:05 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:05.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:05 smithi161 ceph-mon[62387]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:06.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:05 smithi084 ceph-mon[28081]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-28T12:46:06.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:05 smithi084 ceph-mon[28081]: from='osd.8 [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:46:06.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:05 smithi084 ceph-mon[28081]: osdmap e57: 9 total, 8 up, 9 in 2024-04-28T12:46:06.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:05 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:06.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:05 smithi084 ceph-mon[28081]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:46:06.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:05 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:06.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:05 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:06.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:05 smithi084 ceph-mon[28081]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-28T12:46:06.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:06 smithi179 ceph-mon[63029]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-04-28T12:46:06.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:06 smithi179 ceph-mon[63029]: osdmap e58: 9 total, 8 up, 9 in 2024-04-28T12:46:06.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:06 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:06.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:06 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:06.870 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:06 smithi179 ceph-mon[63029]: osd.8 [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948] boot 2024-04-28T12:46:06.871 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:06 smithi179 ceph-mon[63029]: osdmap e59: 9 total, 9 up, 9 in 2024-04-28T12:46:06.871 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:06 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:06.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:06 smithi161 ceph-mon[62387]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-04-28T12:46:06.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:06 smithi161 ceph-mon[62387]: osdmap e58: 9 total, 8 up, 9 in 2024-04-28T12:46:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:06 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:06 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:06 smithi161 ceph-mon[62387]: osd.8 [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948] boot 2024-04-28T12:46:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:06 smithi161 ceph-mon[62387]: osdmap e59: 9 total, 9 up, 9 in 2024-04-28T12:46:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:06 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:07.149 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:06 smithi084 ceph-mon[28081]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-04-28T12:46:07.149 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:06 smithi084 ceph-mon[28081]: osdmap e58: 9 total, 8 up, 9 in 2024-04-28T12:46:07.149 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:06 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:07.149 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:06 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:07.149 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:06 smithi084 ceph-mon[28081]: osd.8 [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948] boot 2024-04-28T12:46:07.149 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:06 smithi084 ceph-mon[28081]: osdmap e59: 9 total, 9 up, 9 in 2024-04-28T12:46:07.149 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:06 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-28T12:46:07.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:07 smithi084 ceph-mon[28081]: purged_snaps scrub starts 2024-04-28T12:46:07.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:07 smithi084 ceph-mon[28081]: purged_snaps scrub ok 2024-04-28T12:46:07.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:07 smithi084 ceph-mon[28081]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:07 smithi161 ceph-mon[62387]: purged_snaps scrub starts 2024-04-28T12:46:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:07 smithi161 ceph-mon[62387]: purged_snaps scrub ok 2024-04-28T12:46:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:07 smithi161 ceph-mon[62387]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:08.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:07 smithi179 ceph-mon[63029]: purged_snaps scrub starts 2024-04-28T12:46:08.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:07 smithi179 ceph-mon[63029]: purged_snaps scrub ok 2024-04-28T12:46:08.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:07 smithi179 ceph-mon[63029]: pgmap v218: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:08.385 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-28T12:46:08.385 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-28T12:46:08.385 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10+0 records in 2024-04-28T12:46:08.385 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2024-04-28T12:46:08.385 INFO:teuthology.orchestra.run.smithi084.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.033843 s, 310 MB/s 2024-04-28T12:46:08.385 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping successful for: 2024-04-28T12:46:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: osdmap e60: 9 total, 9 up, 9 in 2024-04-28T12:46:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: Detected new or changed devices on smithi179 2024-04-28T12:46:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi179 to 5696M 2024-04-28T12:46:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:08 smithi161 ceph-mon[62387]: osdmap e61: 9 total, 9 up, 9 in 2024-04-28T12:46:09.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: osdmap e60: 9 total, 9 up, 9 in 2024-04-28T12:46:09.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: Detected new or changed devices on smithi179 2024-04-28T12:46:09.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:09.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:09.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:09.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:09.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:09.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi179 to 5696M 2024-04-28T12:46:09.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:09.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:09.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:09.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:09.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:08 smithi084 ceph-mon[28081]: osdmap e61: 9 total, 9 up, 9 in 2024-04-28T12:46:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: osdmap e60: 9 total, 9 up, 9 in 2024-04-28T12:46:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: Detected new or changed devices on smithi179 2024-04-28T12:46:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi179 to 5696M 2024-04-28T12:46:09.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:09.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:09.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:09.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:09.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:08 smithi179 ceph-mon[63029]: osdmap e61: 9 total, 9 up, 9 in 2024-04-28T12:46:09.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi084:vg_nvme/lv_1 2024-04-28T12:46:09.888 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:46:09.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:09 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:09.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:09 smithi161 ceph-mon[62387]: osdmap e62: 9 total, 9 up, 9 in 2024-04-28T12:46:09.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:09 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:09.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:09 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:09.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:09 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:09.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:09 smithi161 ceph-mon[62387]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:10.031 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:09 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:10.031 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:09 smithi084 ceph-mon[28081]: osdmap e62: 9 total, 9 up, 9 in 2024-04-28T12:46:10.031 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:09 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:10.031 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:09 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:10.031 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:09 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:10.031 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:09 smithi084 ceph-mon[28081]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:09 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:09 smithi179 ceph-mon[63029]: osdmap e62: 9 total, 9 up, 9 in 2024-04-28T12:46:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:09 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:09 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:09 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:09 smithi179 ceph-mon[63029]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:11.405 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:11 smithi084 ceph-mon[28081]: osdmap e63: 9 total, 9 up, 9 in 2024-04-28T12:46:11.414 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:11 smithi179 ceph-mon[63029]: osdmap e63: 9 total, 9 up, 9 in 2024-04-28T12:46:11.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:11 smithi161 ceph-mon[62387]: osdmap e63: 9 total, 9 up, 9 in 2024-04-28T12:46:12.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.217+0000 7f3ac9600700 1 -- 172.21.15.84:0/39624532 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ac4104680 msgr2=0x7f3ac4104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:12.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.217+0000 7f3ac9600700 1 --2- 172.21.15.84:0/39624532 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ac4104680 0x7f3ac4104ae0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3aac00b0d0 tx=0x7f3aac00b3e0 comp rx=0 tx=0).stop 2024-04-28T12:46:12.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.217+0000 7f3ac9600700 1 -- 172.21.15.84:0/39624532 shutdown_connections 2024-04-28T12:46:12.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.217+0000 7f3ac9600700 1 --2- 172.21.15.84:0/39624532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac4103480 0x7f3ac41038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:12.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.217+0000 7f3ac9600700 1 --2- 172.21.15.84:0/39624532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ac4105020 0x7f3ac410b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:12.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.217+0000 7f3ac9600700 1 --2- 172.21.15.84:0/39624532 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ac4104680 0x7f3ac4104ae0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:12.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.217+0000 7f3ac9600700 1 -- 172.21.15.84:0/39624532 >> 172.21.15.84:0/39624532 conn(0x7f3ac40fea20 msgr2=0x7f3ac4100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:46:12.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.217+0000 7f3ac9600700 1 -- 172.21.15.84:0/39624532 shutdown_connections 2024-04-28T12:46:12.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.217+0000 7f3ac9600700 1 -- 172.21.15.84:0/39624532 wait complete. 2024-04-28T12:46:12.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.218+0000 7f3ac9600700 1 Processor -- start 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.218+0000 7f3ac9600700 1 -- start start 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.218+0000 7f3ac9600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ac4103480 0x7f3ac40771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.218+0000 7f3ac9600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac4104680 0x7f3ac4075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.218+0000 7f3ac9600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ac4105020 0x7f3ac4075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.218+0000 7f3ac9600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac410ef70 con 0x7f3ac4103480 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.218+0000 7f3ac9600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac410edf0 con 0x7f3ac4104680 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.218+0000 7f3ac9600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac410f0f0 con 0x7f3ac4105020 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.219+0000 7f3ac2a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ac4103480 0x7f3ac40771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.219+0000 7f3ac2a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ac4103480 0x7f3ac40771c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37324/0 (socket says 172.21.15.84:37324) 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.219+0000 7f3ac2a00700 1 -- 172.21.15.84:0/2103353181 learned_addr learned my addr 172.21.15.84:0/2103353181 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.219+0000 7f3aba000700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac4104680 0x7f3ac4075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.219+0000 7f3ac3400700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ac4105020 0x7f3ac4075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.219+0000 7f3ac2a00700 1 -- 172.21.15.84:0/2103353181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ac4105020 msgr2=0x7f3ac4075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:12.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.219+0000 7f3ac2a00700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ac4105020 0x7f3ac4075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:12.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.219+0000 7f3ac2a00700 1 -- 172.21.15.84:0/2103353181 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac4104680 msgr2=0x7f3ac4075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:12.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.219+0000 7f3ac2a00700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac4104680 0x7f3ac4075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:12.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.219+0000 7f3ac2a00700 1 -- 172.21.15.84:0/2103353181 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac4076600 con 0x7f3ac4103480 2024-04-28T12:46:12.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.219+0000 7f3aba000700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac4104680 0x7f3ac4075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:46:12.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.220+0000 7f3ac2a00700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ac4103480 0x7f3ac40771c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400ea50 tx=0x7f3ab400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:46:12.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.220+0000 7f3abbe00700 1 -- 172.21.15.84:0/2103353181 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab400cbd0 con 0x7f3ac4103480 2024-04-28T12:46:12.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.220+0000 7f3abbe00700 1 -- 172.21.15.84:0/2103353181 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ab400cd70 con 0x7f3ac4103480 2024-04-28T12:46:12.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.220+0000 7f3ac9600700 1 -- 172.21.15.84:0/2103353181 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac41a3780 con 0x7f3ac4103480 2024-04-28T12:46:12.219 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.220+0000 7f3abbe00700 1 -- 172.21.15.84:0/2103353181 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4018680 con 0x7f3ac4103480 2024-04-28T12:46:12.219 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.220+0000 7f3ac9600700 1 -- 172.21.15.84:0/2103353181 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac41a3c90 con 0x7f3ac4103480 2024-04-28T12:46:12.219 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.221+0000 7f3ac9600700 1 -- 172.21.15.84:0/2103353181 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac406a110 con 0x7f3ac4103480 2024-04-28T12:46:12.220 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.223+0000 7f3abbe00700 1 -- 172.21.15.84:0/2103353181 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3ab401f020 con 0x7f3ac4103480 2024-04-28T12:46:12.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.225+0000 7f3abbe00700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3aa80a3c40 0x7f3aa80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:12.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.225+0000 7f3abbe00700 1 -- 172.21.15.84:0/2103353181 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f3ab40cfa90 con 0x7f3ac4103480 2024-04-28T12:46:12.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.225+0000 7f3aba000700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3aa80a3c40 0x7f3aa80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:12.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.226+0000 7f3aba000700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3aa80a3c40 0x7f3aa80a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009e60 tx=0x7f3aac00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:46:12.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.230+0000 7f3abbe00700 1 -- 172.21.15.84:0/2103353181 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3ab409cd80 con 0x7f3ac4103480 2024-04-28T12:46:12.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:12 smithi179 ceph-mon[63029]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:46:12.416 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:12 smithi084 ceph-mon[28081]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:46:12.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:12.466+0000 7f3ac9600700 1 -- 172.21.15.84:0/2103353181 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3ac4063860 con 0x7f3aa80a3c40 2024-04-28T12:46:12.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:12 smithi161 ceph-mon[62387]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:46:13.405 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:13 smithi084 ceph-mon[28081]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:46:13.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:46:13.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:46:13.406 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:13.414 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:13 smithi179 ceph-mon[63029]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:46:13.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:46:13.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:46:13.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:13.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:13 smithi161 ceph-mon[62387]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:46:13.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:46:13.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:46:13.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:14.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:14 smithi179 ceph-mon[63029]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-28T12:46:14.416 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:14 smithi084 ceph-mon[28081]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-28T12:46:14.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:14 smithi161 ceph-mon[62387]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-28T12:46:16.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:16 smithi179 ceph-mon[63029]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-28T12:46:16.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:16 smithi084 ceph-mon[28081]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-28T12:46:16.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:16 smithi161 ceph-mon[62387]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-28T12:46:17.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:17 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1614871978' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bc308bb-cc85-40b4-9af0-7dec273748f7"}]: dispatch 2024-04-28T12:46:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:17 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1614871978' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7bc308bb-cc85-40b4-9af0-7dec273748f7"}]': finished 2024-04-28T12:46:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:17 smithi179 ceph-mon[63029]: osdmap e64: 10 total, 9 up, 10 in 2024-04-28T12:46:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:17 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:17.924 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:17 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/354840701' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:46:17.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:17 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1614871978' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bc308bb-cc85-40b4-9af0-7dec273748f7"}]: dispatch 2024-04-28T12:46:17.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:17 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1614871978' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7bc308bb-cc85-40b4-9af0-7dec273748f7"}]': finished 2024-04-28T12:46:17.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:17 smithi084 ceph-mon[28081]: osdmap e64: 10 total, 9 up, 10 in 2024-04-28T12:46:17.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:17 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:17.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:17 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/354840701' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:46:17.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:17 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1614871978' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bc308bb-cc85-40b4-9af0-7dec273748f7"}]: dispatch 2024-04-28T12:46:17.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:17 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1614871978' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7bc308bb-cc85-40b4-9af0-7dec273748f7"}]': finished 2024-04-28T12:46:17.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:17 smithi161 ceph-mon[62387]: osdmap e64: 10 total, 9 up, 10 in 2024-04-28T12:46:17.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:17 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:17.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:17 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/354840701' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:46:18.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:18 smithi179 ceph-mon[63029]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T12:46:18.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:18 smithi084 ceph-mon[28081]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T12:46:18.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:18 smithi161 ceph-mon[62387]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-28T12:46:20.884 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:20 smithi084 ceph-mon[28081]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-28T12:46:20.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:20 smithi179 ceph-mon[63029]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-28T12:46:20.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:20 smithi161 ceph-mon[62387]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-28T12:46:22.902 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:22 smithi084 ceph-mon[28081]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:22.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:22 smithi179 ceph-mon[63029]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:22.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:22 smithi161 ceph-mon[62387]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:24.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:24 smithi084 ceph-mon[28081]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:24.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:24 smithi179 ceph-mon[63029]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:24.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:24 smithi161 ceph-mon[62387]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:26.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:26 smithi179 ceph-mon[63029]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:26.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-28T12:46:26.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:26.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:26 smithi084 ceph-mon[28081]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:26.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-28T12:46:26.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:26.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:26 smithi161 ceph-mon[62387]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:26.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-28T12:46:26.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:27.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:27 smithi179 ceph-mon[63029]: Deploying daemon osd.9 on smithi084 2024-04-28T12:46:27.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:27 smithi084 ceph-mon[28081]: Deploying daemon osd.9 on smithi084 2024-04-28T12:46:27.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:27 smithi161 ceph-mon[62387]: Deploying daemon osd.9 on smithi084 2024-04-28T12:46:28.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:28 smithi179 ceph-mon[63029]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:28.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:28 smithi084 ceph-mon[28081]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:28.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:28 smithi161 ceph-mon[62387]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:29.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:29 smithi179 ceph-mon[63029]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:29.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:29 smithi084 ceph-mon[28081]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:29.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:29 smithi161 ceph-mon[62387]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:32 smithi179 ceph-mon[63029]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:32.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:32 smithi084 ceph-mon[28081]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:32.966 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:32 smithi161 ceph-mon[62387]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:34 smithi179 ceph-mon[63029]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:34.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:34 smithi084 ceph-mon[28081]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:34.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:34 smithi161 ceph-mon[62387]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:36.416 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:36 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:36.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:36 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:36.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:36 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:36.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:36 smithi084 ceph-mon[28081]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:36.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:36 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:36.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:36 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:36.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:36 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:36.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:36 smithi179 ceph-mon[63029]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:36.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:36 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:36.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:36 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:36.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:36 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:36.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:36 smithi161 ceph-mon[62387]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:38.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:38 smithi084 ceph-mon[28081]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:38 smithi179 ceph-mon[63029]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:38.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:38 smithi161 ceph-mon[62387]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:39.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:39 smithi179 ceph-mon[63029]: from='osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-28T12:46:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:39.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:39 smithi084 ceph-mon[28081]: from='osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-28T12:46:39.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:39.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:39.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:39 smithi161 ceph-mon[62387]: from='osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-28T12:46:39.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:39 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:39.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:39 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:40.677 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.680+0000 7f3abbe00700 1 -- 172.21.15.84:0/2103353181 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3ac4063860 con 0x7f3aa80a3c40 2024-04-28T12:46:40.678 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 9 on host 'smithi084' 2024-04-28T12:46:40.680 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.684+0000 7f3ac9600700 1 -- 172.21.15.84:0/2103353181 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3aa80a3c40 msgr2=0x7f3aa80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:40.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.684+0000 7f3ac9600700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3aa80a3c40 0x7f3aa80a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009e60 tx=0x7f3aac00b040 comp rx=0 tx=0).stop 2024-04-28T12:46:40.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.684+0000 7f3ac9600700 1 -- 172.21.15.84:0/2103353181 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ac4103480 msgr2=0x7f3ac40771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:40.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.684+0000 7f3ac9600700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ac4103480 0x7f3ac40771c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400ea50 tx=0x7f3ab400ed60 comp rx=0 tx=0).stop 2024-04-28T12:46:40.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.684+0000 7f3ac9600700 1 -- 172.21.15.84:0/2103353181 shutdown_connections 2024-04-28T12:46:40.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.684+0000 7f3ac9600700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3ac4105020 0x7f3ac4075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:40.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.684+0000 7f3ac9600700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ac4103480 0x7f3ac40771c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:40.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.684+0000 7f3ac9600700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3aa80a3c40 0x7f3aa80a6100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:40.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.684+0000 7f3ac9600700 1 --2- 172.21.15.84:0/2103353181 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ac4104680 0x7f3ac4075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:40.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.684+0000 7f3ac9600700 1 -- 172.21.15.84:0/2103353181 >> 172.21.15.84:0/2103353181 conn(0x7f3ac40fea20 msgr2=0x7f3ac41079f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:46:40.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.685+0000 7f3ac9600700 1 -- 172.21.15.84:0/2103353181 shutdown_connections 2024-04-28T12:46:40.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:46:40.685+0000 7f3ac9600700 1 -- 172.21.15.84:0/2103353181 wait complete. 2024-04-28T12:46:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:40 smithi179 ceph-mon[63029]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:40 smithi179 ceph-mon[63029]: from='osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-28T12:46:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:40 smithi179 ceph-mon[63029]: osdmap e65: 10 total, 9 up, 10 in 2024-04-28T12:46:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:40 smithi179 ceph-mon[63029]: from='osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-04-28T12:46:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:40 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:40.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:40 smithi084 ceph-mon[28081]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:40.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:40 smithi084 ceph-mon[28081]: from='osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-28T12:46:40.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:40 smithi084 ceph-mon[28081]: osdmap e65: 10 total, 9 up, 10 in 2024-04-28T12:46:40.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:40 smithi084 ceph-mon[28081]: from='osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-04-28T12:46:40.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:40 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:40.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:40 smithi161 ceph-mon[62387]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:40.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:40 smithi161 ceph-mon[62387]: from='osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-28T12:46:40.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:40 smithi161 ceph-mon[62387]: osdmap e65: 10 total, 9 up, 10 in 2024-04-28T12:46:40.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:40 smithi161 ceph-mon[62387]: from='osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-04-28T12:46:40.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:40 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:41.805 DEBUG:teuthology.orchestra.run.smithi084:osd.9> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.9.service 2024-04-28T12:46:41.810 INFO:tasks.cephadm:Deploying osd.10 on smithi161 with /dev/vg_nvme/lv_1... 2024-04-28T12:46:41.810 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-28T12:46:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:41 smithi179 ceph-mon[63029]: from='osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-04-28T12:46:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:41 smithi179 ceph-mon[63029]: osdmap e66: 10 total, 9 up, 10 in 2024-04-28T12:46:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:41 smithi179 ceph-mon[63029]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:41.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:41 smithi084 ceph-mon[28081]: from='osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-04-28T12:46:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:41 smithi084 ceph-mon[28081]: osdmap e66: 10 total, 9 up, 10 in 2024-04-28T12:46:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:41.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:41 smithi084 ceph-mon[28081]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:41.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:41 smithi161 ceph-mon[62387]: from='osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-04-28T12:46:41.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:41 smithi161 ceph-mon[62387]: osdmap e66: 10 total, 9 up, 10 in 2024-04-28T12:46:41.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:41.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:41.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:41.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:41.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:41.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:41.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:41.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:46:41.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:41 smithi161 ceph-mon[62387]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-04-28T12:46:42.058 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:46:42.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:42 smithi179 ceph-mon[63029]: purged_snaps scrub starts 2024-04-28T12:46:42.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:42 smithi179 ceph-mon[63029]: purged_snaps scrub ok 2024-04-28T12:46:42.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:42.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:42 smithi179 ceph-mon[63029]: osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312] boot 2024-04-28T12:46:42.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:42 smithi179 ceph-mon[63029]: osdmap e67: 10 total, 10 up, 10 in 2024-04-28T12:46:42.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:42.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:42 smithi084 ceph-mon[28081]: purged_snaps scrub starts 2024-04-28T12:46:42.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:42 smithi084 ceph-mon[28081]: purged_snaps scrub ok 2024-04-28T12:46:42.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:42.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:42 smithi084 ceph-mon[28081]: osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312] boot 2024-04-28T12:46:42.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:42 smithi084 ceph-mon[28081]: osdmap e67: 10 total, 10 up, 10 in 2024-04-28T12:46:42.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:42.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:42 smithi161 ceph-mon[62387]: purged_snaps scrub starts 2024-04-28T12:46:42.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:42 smithi161 ceph-mon[62387]: purged_snaps scrub ok 2024-04-28T12:46:42.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:42.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:42 smithi161 ceph-mon[62387]: osd.9 [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312] boot 2024-04-28T12:46:42.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:42 smithi161 ceph-mon[62387]: osdmap e67: 10 total, 10 up, 10 in 2024-04-28T12:46:42.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-28T12:46:43.966 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:43 smithi161 ceph-mon[62387]: osdmap e68: 10 total, 10 up, 10 in 2024-04-28T12:46:43.966 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:43 smithi161 ceph-mon[62387]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:44.007 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:43 smithi084 ceph-mon[28081]: osdmap e68: 10 total, 10 up, 10 in 2024-04-28T12:46:44.007 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:43 smithi084 ceph-mon[28081]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:44.028 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-28T12:46:44.028 INFO:teuthology.orchestra.run.smithi161.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-28T12:46:44.028 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10+0 records in 2024-04-28T12:46:44.028 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2024-04-28T12:46:44.028 INFO:teuthology.orchestra.run.smithi161.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011376 s, 922 MB/s 2024-04-28T12:46:44.029 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 2024-04-28T12:46:44.029 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping successful for: 2024-04-28T12:46:44.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:43 smithi179 ceph-mon[63029]: osdmap e68: 10 total, 10 up, 10 in 2024-04-28T12:46:44.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:43 smithi179 ceph-mon[63029]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:44.676 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi161:vg_nvme/lv_1 2024-04-28T12:46:44.922 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.b/config 2024-04-28T12:46:46.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.759+0000 7febd9dd0700 1 -- 172.21.15.161:0/3951434075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febd4106850 msgr2=0x7febd4108c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:46.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.759+0000 7febd9dd0700 1 --2- 172.21.15.161:0/3951434075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febd4106850 0x7febd4108c40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7febc8009990 tx=0x7febc8009ca0 comp rx=0 tx=0).stop 2024-04-28T12:46:46.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.760+0000 7febd9dd0700 1 -- 172.21.15.161:0/3951434075 shutdown_connections 2024-04-28T12:46:46.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.760+0000 7febd9dd0700 1 --2- 172.21.15.161:0/3951434075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febd4109180 0x7febd410b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:46.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.760+0000 7febd9dd0700 1 --2- 172.21.15.161:0/3951434075 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7febd4103f20 0x7febd4106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:46.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.760+0000 7febd9dd0700 1 --2- 172.21.15.161:0/3951434075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febd4106850 0x7febd4108c40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:46.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.760+0000 7febd9dd0700 1 -- 172.21.15.161:0/3951434075 >> 172.21.15.161:0/3951434075 conn(0x7febd40fd950 msgr2=0x7febd40ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:46:46.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.760+0000 7febd9dd0700 1 -- 172.21.15.161:0/3951434075 shutdown_connections 2024-04-28T12:46:46.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.760+0000 7febd9dd0700 1 -- 172.21.15.161:0/3951434075 wait complete. 2024-04-28T12:46:46.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.760+0000 7febd9dd0700 1 Processor -- start 2024-04-28T12:46:46.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.760+0000 7febd9dd0700 1 -- start start 2024-04-28T12:46:46.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.760+0000 7febd9dd0700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febd4103f20 0x7febd4103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:46.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.760+0000 7febd9dd0700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febd4106850 0x7febd4101ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.760+0000 7febd9dd0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7febd4109180 0x7febd4102410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd9dd0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7febd410efe0 con 0x7febd4109180 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd9dd0700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7febd410ee60 con 0x7febd4106850 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd9dd0700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7febd410f160 con 0x7febd4103f20 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd2ffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febd4106850 0x7febd4101ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd2ffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febd4106850 0x7febd4101ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55698/0 (socket says 172.21.15.161:55698) 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd2ffd700 1 -- 172.21.15.161:0/2755485522 learned_addr learned my addr 172.21.15.161:0/2755485522 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd37fe700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febd4103f20 0x7febd4103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd2ffd700 1 -- 172.21.15.161:0/2755485522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febd4103f20 msgr2=0x7febd4103880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd2ffd700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febd4103f20 0x7febd4103880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd2ffd700 1 -- 172.21.15.161:0/2755485522 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7febd4109180 msgr2=0x7febd4102410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd2ffd700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7febd4109180 0x7febd4102410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd2ffd700 1 -- 172.21.15.161:0/2755485522 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd4102cc0 con 0x7febd4106850 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd2ffd700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febd4106850 0x7febd4101ed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7febc80098e0 tx=0x7febc80057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:46:46.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd0ff9700 1 -- 172.21.15.161:0/2755485522 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc8026070 con 0x7febd4106850 2024-04-28T12:46:46.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd0ff9700 1 -- 172.21.15.161:0/2755485522 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febc8004080 con 0x7febd4106850 2024-04-28T12:46:46.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd9dd0700 1 -- 172.21.15.161:0/2755485522 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd41a37c0 con 0x7febd4106850 2024-04-28T12:46:46.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd0ff9700 1 -- 172.21.15.161:0/2755485522 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc800fe60 con 0x7febd4106850 2024-04-28T12:46:46.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.761+0000 7febd9dd0700 1 -- 172.21.15.161:0/2755485522 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd41a3c90 con 0x7febd4106850 2024-04-28T12:46:46.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.762+0000 7febd9dd0700 1 -- 172.21.15.161:0/2755485522 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febd406a110 con 0x7febd4106850 2024-04-28T12:46:46.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.763+0000 7febd0ff9700 1 -- 172.21.15.161:0/2755485522 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7febc8029070 con 0x7febd4106850 2024-04-28T12:46:46.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.764+0000 7febd0ff9700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7febbc0a3c40 0x7febbc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:46:46.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.764+0000 7febd0ff9700 1 -- 172.21.15.161:0/2755485522 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7febc80d0d60 con 0x7febd4106850 2024-04-28T12:46:46.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.764+0000 7febd37fe700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7febbc0a3c40 0x7febbc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:46:46.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.765+0000 7febd37fe700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7febbc0a3c40 0x7febbc0a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7febc4009a10 tx=0x7febc4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:46:46.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.767+0000 7febd0ff9700 1 -- 172.21.15.161:0/2755485522 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7febc8030080 con 0x7febd4106850 2024-04-28T12:46:46.868 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:46 smithi179 ceph-mon[63029]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:46.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:46 smithi084 ceph-mon[28081]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:46.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:46:46.979+0000 7febd9dd0700 1 -- 172.21.15.161:0/2755485522 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7febd4063920 con 0x7febbc0a3c40 2024-04-28T12:46:46.975 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:46 smithi161 ceph-mon[62387]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:47.872 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: Detected new or changed devices on smithi084 2024-04-28T12:46:47.872 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:47.872 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:47.872 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:47.872 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:47.872 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:47.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:47.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi084 to 4272M 2024-04-28T12:46:47.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:47.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:47.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:47.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:47.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:46:47.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:46:47.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:46:47.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:47.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:47 smithi161 ceph-mon[62387]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:48.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: Detected new or changed devices on smithi084 2024-04-28T12:46:48.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:48.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:48.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi084 to 4272M 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:47 smithi179 ceph-mon[63029]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: Detected new or changed devices on smithi084 2024-04-28T12:46:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:46:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi084 to 4272M 2024-04-28T12:46:48.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:48.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:48.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:46:48.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:46:48.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:46:48.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:46:48.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:46:48.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:48.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:47 smithi084 ceph-mon[28081]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:50.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:50 smithi179 ceph-mon[63029]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:50.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:50 smithi179 ceph-mon[63029]: from='client.? 172.21.15.161:0/2894713316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f25ff51-6d47-4313-a7af-4be37046c627"}]: dispatch 2024-04-28T12:46:50.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:50 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f25ff51-6d47-4313-a7af-4be37046c627"}]: dispatch 2024-04-28T12:46:50.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:50 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f25ff51-6d47-4313-a7af-4be37046c627"}]': finished 2024-04-28T12:46:50.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:50 smithi179 ceph-mon[63029]: osdmap e69: 11 total, 10 up, 11 in 2024-04-28T12:46:50.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:46:50.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:50 smithi179 ceph-mon[63029]: from='client.? 172.21.15.161:0/1772286571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:46:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:50 smithi084 ceph-mon[28081]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:50 smithi084 ceph-mon[28081]: from='client.? 172.21.15.161:0/2894713316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f25ff51-6d47-4313-a7af-4be37046c627"}]: dispatch 2024-04-28T12:46:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:50 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f25ff51-6d47-4313-a7af-4be37046c627"}]: dispatch 2024-04-28T12:46:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:50 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f25ff51-6d47-4313-a7af-4be37046c627"}]': finished 2024-04-28T12:46:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:50 smithi084 ceph-mon[28081]: osdmap e69: 11 total, 10 up, 11 in 2024-04-28T12:46:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:46:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:50 smithi084 ceph-mon[28081]: from='client.? 172.21.15.161:0/1772286571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:46:50.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:50 smithi161 ceph-mon[62387]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:50.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:50 smithi161 ceph-mon[62387]: from='client.? 172.21.15.161:0/2894713316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f25ff51-6d47-4313-a7af-4be37046c627"}]: dispatch 2024-04-28T12:46:50.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:50 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f25ff51-6d47-4313-a7af-4be37046c627"}]: dispatch 2024-04-28T12:46:50.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:50 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f25ff51-6d47-4313-a7af-4be37046c627"}]': finished 2024-04-28T12:46:50.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:50 smithi161 ceph-mon[62387]: osdmap e69: 11 total, 10 up, 11 in 2024-04-28T12:46:50.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:46:50.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:50 smithi161 ceph-mon[62387]: from='client.? 172.21.15.161:0/1772286571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:46:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:52 smithi179 ceph-mon[63029]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:52.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:52 smithi084 ceph-mon[28081]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:52.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:52 smithi161 ceph-mon[62387]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:54 smithi161 ceph-mon[62387]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:54.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:54 smithi179 ceph-mon[63029]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:54.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:54 smithi084 ceph-mon[28081]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:56.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:56 smithi179 ceph-mon[63029]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:56.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:56 smithi084 ceph-mon[28081]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:56.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:56 smithi161 ceph-mon[62387]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:57.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:57 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-28T12:46:57.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:57 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:57.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:57 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-28T12:46:57.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:57 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:57.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:57 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-28T12:46:57.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:57 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:46:58.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:58 smithi179 ceph-mon[63029]: Deploying daemon osd.10 on smithi161 2024-04-28T12:46:58.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:58 smithi179 ceph-mon[63029]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:58.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:58 smithi084 ceph-mon[28081]: Deploying daemon osd.10 on smithi161 2024-04-28T12:46:58.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:58 smithi084 ceph-mon[28081]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:58.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:58 smithi161 ceph-mon[62387]: Deploying daemon osd.10 on smithi161 2024-04-28T12:46:58.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:58 smithi161 ceph-mon[62387]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:46:59.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:46:59 smithi161 ceph-mon[62387]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:00.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:46:59 smithi179 ceph-mon[63029]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:00.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:46:59 smithi084 ceph-mon[28081]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:02.783 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:02 smithi161 ceph-mon[62387]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:02.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:02 smithi179 ceph-mon[63029]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:02.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:02 smithi084 ceph-mon[28081]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:04.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:04 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:04.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:04 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:04.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:04 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:04.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:04 smithi161 ceph-mon[62387]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:04.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:04 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:04.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:04 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:04.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:04 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:04.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:04 smithi179 ceph-mon[63029]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:04.666 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:04 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:04.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:04 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:04.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:04 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:04.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:04 smithi084 ceph-mon[28081]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:06.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.462+0000 7febd0ff9700 1 -- 172.21.15.161:0/2755485522 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7febd4063920 con 0x7febbc0a3c40 2024-04-28T12:47:06.460 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 10 on host 'smithi161' 2024-04-28T12:47:06.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.464+0000 7febd9dd0700 1 -- 172.21.15.161:0/2755485522 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7febbc0a3c40 msgr2=0x7febbc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:06.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.464+0000 7febd9dd0700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7febbc0a3c40 0x7febbc0a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7febc4009a10 tx=0x7febc4009190 comp rx=0 tx=0).stop 2024-04-28T12:47:06.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.464+0000 7febd9dd0700 1 -- 172.21.15.161:0/2755485522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febd4106850 msgr2=0x7febd4101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:06.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.464+0000 7febd9dd0700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febd4106850 0x7febd4101ed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7febc80098e0 tx=0x7febc80057c0 comp rx=0 tx=0).stop 2024-04-28T12:47:06.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.464+0000 7febd9dd0700 1 -- 172.21.15.161:0/2755485522 shutdown_connections 2024-04-28T12:47:06.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.464+0000 7febd9dd0700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7febd4109180 0x7febd4102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:06.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.464+0000 7febd9dd0700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febd4103f20 0x7febd4103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:06.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.464+0000 7febd9dd0700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7febbc0a3c40 0x7febbc0a6100 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:06.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.464+0000 7febd9dd0700 1 --2- 172.21.15.161:0/2755485522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febd4106850 0x7febd4101ed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:06.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.464+0000 7febd9dd0700 1 -- 172.21.15.161:0/2755485522 >> 172.21.15.161:0/2755485522 conn(0x7febd40fd950 msgr2=0x7febd4104860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:06.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.465+0000 7febd9dd0700 1 -- 172.21.15.161:0/2755485522 shutdown_connections 2024-04-28T12:47:06.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-28T12:47:06.465+0000 7febd9dd0700 1 -- 172.21.15.161:0/2755485522 wait complete. 2024-04-28T12:47:06.701 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:06 smithi161 ceph-mon[62387]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:06.701 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:06 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:06.701 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:06 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:06 smithi179 ceph-mon[63029]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:06 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:06 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:06.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:06 smithi084 ceph-mon[28081]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:06.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:06 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:06.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:06 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:07.362 DEBUG:teuthology.orchestra.run.smithi161:osd.10> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.10.service 2024-04-28T12:47:07.364 INFO:tasks.cephadm:Deploying osd.11 on smithi179 with /dev/vg_nvme/lv_1... 2024-04-28T12:47:07.364 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-28T12:47:07.615 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:47:07.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:07 smithi179 ceph-mon[63029]: from='osd.10 [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T12:47:07.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:07 smithi179 ceph-mon[63029]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T12:47:07.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:07 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:07.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:07 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:07.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:07 smithi084 ceph-mon[28081]: from='osd.10 [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T12:47:07.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:07 smithi084 ceph-mon[28081]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T12:47:07.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:07 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:07.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:07 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:07 smithi161 ceph-mon[62387]: from='osd.10 [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T12:47:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:07 smithi161 ceph-mon[62387]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-28T12:47:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:07 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:07 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:08.888 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:08 smithi179 ceph-mon[63029]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:08.888 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:08 smithi179 ceph-mon[63029]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-28T12:47:08.888 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:08 smithi179 ceph-mon[63029]: from='osd.10 [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:47:08.888 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:08 smithi179 ceph-mon[63029]: osdmap e70: 11 total, 10 up, 11 in 2024-04-28T12:47:08.888 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:08.889 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:08 smithi179 ceph-mon[63029]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:47:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:08 smithi084 ceph-mon[28081]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:08 smithi084 ceph-mon[28081]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-28T12:47:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:08 smithi084 ceph-mon[28081]: from='osd.10 [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:47:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:08 smithi084 ceph-mon[28081]: osdmap e70: 11 total, 10 up, 11 in 2024-04-28T12:47:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:08 smithi084 ceph-mon[28081]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:47:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:08 smithi161 ceph-mon[62387]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:08 smithi161 ceph-mon[62387]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-28T12:47:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:08 smithi161 ceph-mon[62387]: from='osd.10 [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:47:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:08 smithi161 ceph-mon[62387]: osdmap e70: 11 total, 10 up, 11 in 2024-04-28T12:47:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:08 smithi161 ceph-mon[62387]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-04-28T12:47:08.993 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 12:47:08 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T12:47:08.642+0000 7ffbc604d700 -1 osd.10 0 waiting for initial osdmap 2024-04-28T12:47:08.993 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 12:47:08 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T12:47:08.658+0000 7ffbc1e40700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:47:09.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:09 smithi161 ceph-mon[62387]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-04-28T12:47:09.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:09 smithi161 ceph-mon[62387]: osdmap e71: 11 total, 10 up, 11 in 2024-04-28T12:47:09.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:09 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:09.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:09 smithi161 ceph-mon[62387]: osd.10 [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178] boot 2024-04-28T12:47:09.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:09 smithi161 ceph-mon[62387]: osdmap e72: 11 total, 11 up, 11 in 2024-04-28T12:47:09.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:09 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:09.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:09 smithi161 ceph-mon[62387]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:09.779 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-28T12:47:09.779 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-28T12:47:09.779 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-04-28T12:47:09.779 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-04-28T12:47:09.779 INFO:teuthology.orchestra.run.smithi179.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011347 s, 924 MB/s 2024-04-28T12:47:09.779 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-04-28T12:47:09.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:09 smithi179 ceph-mon[63029]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-04-28T12:47:09.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:09 smithi179 ceph-mon[63029]: osdmap e71: 11 total, 10 up, 11 in 2024-04-28T12:47:09.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:09 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:09.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:09 smithi179 ceph-mon[63029]: osd.10 [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178] boot 2024-04-28T12:47:09.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:09 smithi179 ceph-mon[63029]: osdmap e72: 11 total, 11 up, 11 in 2024-04-28T12:47:09.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:09 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:09.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:09 smithi179 ceph-mon[63029]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:09.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:09 smithi084 ceph-mon[28081]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-04-28T12:47:09.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:09 smithi084 ceph-mon[28081]: osdmap e71: 11 total, 10 up, 11 in 2024-04-28T12:47:09.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:09 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:09.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:09 smithi084 ceph-mon[28081]: osd.10 [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178] boot 2024-04-28T12:47:09.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:09 smithi084 ceph-mon[28081]: osdmap e72: 11 total, 11 up, 11 in 2024-04-28T12:47:09.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:09 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-28T12:47:09.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:09 smithi084 ceph-mon[28081]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-28T12:47:10.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi179:vg_nvme/lv_1 2024-04-28T12:47:10.748 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:47:10.781 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:10 smithi179 ceph-mon[63029]: purged_snaps scrub starts 2024-04-28T12:47:10.781 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:10 smithi179 ceph-mon[63029]: purged_snaps scrub ok 2024-04-28T12:47:10.781 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:10 smithi179 ceph-mon[63029]: osdmap e73: 11 total, 11 up, 11 in 2024-04-28T12:47:10.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:10 smithi161 ceph-mon[62387]: purged_snaps scrub starts 2024-04-28T12:47:10.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:10 smithi161 ceph-mon[62387]: purged_snaps scrub ok 2024-04-28T12:47:10.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:10 smithi161 ceph-mon[62387]: osdmap e73: 11 total, 11 up, 11 in 2024-04-28T12:47:10.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:10 smithi084 ceph-mon[28081]: purged_snaps scrub starts 2024-04-28T12:47:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:10 smithi084 ceph-mon[28081]: purged_snaps scrub ok 2024-04-28T12:47:10.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:10 smithi084 ceph-mon[28081]: osdmap e73: 11 total, 11 up, 11 in 2024-04-28T12:47:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: Detected new or changed devices on smithi161 2024-04-28T12:47:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi161 to 4272M 2024-04-28T12:47:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:12.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:12.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:12.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:12.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: osdmap e74: 11 total, 11 up, 11 in 2024-04-28T12:47:12.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:12.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:11 smithi084 ceph-mon[28081]: pgmap v265: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-04-28T12:47:12.223 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: Detected new or changed devices on smithi161 2024-04-28T12:47:12.223 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:12.223 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:12.223 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:12.223 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:12.223 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:12.223 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:12.223 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi161 to 4272M 2024-04-28T12:47:12.223 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:12.223 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:12.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:12.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:12.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: osdmap e74: 11 total, 11 up, 11 in 2024-04-28T12:47:12.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:12.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:11 smithi179 ceph-mon[63029]: pgmap v265: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-04-28T12:47:12.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: Detected new or changed devices on smithi161 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi161 to 4272M 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: osdmap e74: 11 total, 11 up, 11 in 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:11 smithi161 ceph-mon[62387]: pgmap v265: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-04-28T12:47:12.474 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.458+0000 7ff92f3c4700 1 -- 172.21.15.179:0/4091341180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff928105040 msgr2=0x7ff92810b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:12.474 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.458+0000 7ff92f3c4700 1 --2- 172.21.15.179:0/4091341180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff928105040 0x7ff92810b8f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff92000b0a0 tx=0x7ff92000b3b0 comp rx=0 tx=0).stop 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.458+0000 7ff92f3c4700 1 -- 172.21.15.179:0/4091341180 shutdown_connections 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.458+0000 7ff92f3c4700 1 --2- 172.21.15.179:0/4091341180 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff928105040 0x7ff92810b8f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.458+0000 7ff92f3c4700 1 --2- 172.21.15.179:0/4091341180 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9281034a0 0x7ff9281038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.458+0000 7ff92f3c4700 1 --2- 172.21.15.179:0/4091341180 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9281046a0 0x7ff928104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92f3c4700 1 -- 172.21.15.179:0/4091341180 >> 172.21.15.179:0/4091341180 conn(0x7ff9280fea40 msgr2=0x7ff928100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92f3c4700 1 -- 172.21.15.179:0/4091341180 shutdown_connections 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92f3c4700 1 -- 172.21.15.179:0/4091341180 wait complete. 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92f3c4700 1 Processor -- start 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92f3c4700 1 -- start start 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92f3c4700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9281034a0 0x7ff9281995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92f3c4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9281046a0 0x7ff928199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92f3c4700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff928105040 0x7ff9281a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:12.475 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92f3c4700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff92810efc0 con 0x7ff9281046a0 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92f3c4700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff92810ee40 con 0x7ff928105040 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92f3c4700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff92810f140 con 0x7ff9281034a0 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92d160700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9281034a0 0x7ff9281995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92d160700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9281034a0 0x7ff9281995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:42162/0 (socket says 172.21.15.179:42162) 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.459+0000 7ff92d160700 1 -- 172.21.15.179:0/1076503810 learned_addr learned my addr 172.21.15.179:0/1076503810 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.460+0000 7ff92d961700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff928105040 0x7ff9281a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.460+0000 7ff92d160700 1 -- 172.21.15.179:0/1076503810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff928105040 msgr2=0x7ff9281a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.460+0000 7ff92d160700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff928105040 0x7ff9281a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.460+0000 7ff92d160700 1 -- 172.21.15.179:0/1076503810 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9281046a0 msgr2=0x7ff928199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.460+0000 7ff92d160700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9281046a0 0x7ff928199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.460+0000 7ff92d160700 1 -- 172.21.15.179:0/1076503810 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9281a1230 con 0x7ff9281034a0 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.460+0000 7ff92d160700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9281034a0 0x7ff9281995d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff92400b530 tx=0x7ff92400b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:12.476 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.460+0000 7ff91e7fc700 1 -- 172.21.15.179:0/1076503810 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff924015020 con 0x7ff9281034a0 2024-04-28T12:47:12.477 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.460+0000 7ff91e7fc700 1 -- 172.21.15.179:0/1076503810 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff924011ce0 con 0x7ff9281034a0 2024-04-28T12:47:12.477 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.460+0000 7ff91e7fc700 1 -- 172.21.15.179:0/1076503810 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff924010d60 con 0x7ff9281034a0 2024-04-28T12:47:12.477 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.460+0000 7ff92f3c4700 1 -- 172.21.15.179:0/1076503810 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9281a1520 con 0x7ff9281034a0 2024-04-28T12:47:12.477 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.460+0000 7ff92f3c4700 1 -- 172.21.15.179:0/1076503810 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9281a1b30 con 0x7ff9281034a0 2024-04-28T12:47:12.477 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.461+0000 7ff92f3c4700 1 -- 172.21.15.179:0/1076503810 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff92806a110 con 0x7ff9281034a0 2024-04-28T12:47:12.477 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.462+0000 7ff91e7fc700 1 -- 172.21.15.179:0/1076503810 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff924018020 con 0x7ff9281034a0 2024-04-28T12:47:12.477 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.463+0000 7ff91e7fc700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff9140a3cc0 0x7ff9140a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:12.477 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.463+0000 7ff91e7fc700 1 -- 172.21.15.179:0/1076503810 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7ff9240cfa10 con 0x7ff9281034a0 2024-04-28T12:47:12.477 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.463+0000 7ff92c95f700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff9140a3cc0 0x7ff9140a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:12.477 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.464+0000 7ff92c95f700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff9140a3cc0 0x7ff9140a6180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff918005950 tx=0x7ff9180058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:12.477 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.467+0000 7ff91e7fc700 1 -- 172.21.15.179:0/1076503810 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff92401f020 con 0x7ff9281034a0 2024-04-28T12:47:12.669 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:12.673+0000 7ff92f3c4700 1 -- 172.21.15.179:0/1076503810 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff928063860 con 0x7ff9140a3cc0 2024-04-28T12:47:13.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:13.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:13.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:13 smithi179 ceph-mon[63029]: osdmap e75: 11 total, 11 up, 11 in 2024-04-28T12:47:13.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:13.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:47:13.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:47:13.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:13 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:13.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:13.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:13.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:13 smithi084 ceph-mon[28081]: osdmap e75: 11 total, 11 up, 11 in 2024-04-28T12:47:13.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:13.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:47:13.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:47:13.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:13 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:13.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:13.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:13.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:13 smithi161 ceph-mon[62387]: osdmap e75: 11 total, 11 up, 11 in 2024-04-28T12:47:13.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:13.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-28T12:47:13.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-28T12:47:13.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:13 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:14 smithi161 ceph-mon[62387]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:47:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:14 smithi161 ceph-mon[62387]: osdmap e76: 11 total, 11 up, 11 in 2024-04-28T12:47:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:14 smithi161 ceph-mon[62387]: pgmap v268: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:47:14.507 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:14 smithi179 ceph-mon[63029]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:47:14.507 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:14 smithi179 ceph-mon[63029]: osdmap e76: 11 total, 11 up, 11 in 2024-04-28T12:47:14.507 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:14 smithi179 ceph-mon[63029]: pgmap v268: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:47:14.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:14 smithi084 ceph-mon[28081]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:47:14.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:14 smithi084 ceph-mon[28081]: osdmap e76: 11 total, 11 up, 11 in 2024-04-28T12:47:14.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:14 smithi084 ceph-mon[28081]: pgmap v268: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-28T12:47:15.850 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:15 smithi179 ceph-mon[63029]: from='client.? 172.21.15.179:0/2114104478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6babaf1e-dfe0-4f33-a484-ba66a566d5d3"}]: dispatch 2024-04-28T12:47:15.850 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:15 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6babaf1e-dfe0-4f33-a484-ba66a566d5d3"}]: dispatch 2024-04-28T12:47:15.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:15 smithi179 ceph-mon[63029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6babaf1e-dfe0-4f33-a484-ba66a566d5d3"}]': finished 2024-04-28T12:47:15.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:15 smithi179 ceph-mon[63029]: osdmap e77: 12 total, 11 up, 12 in 2024-04-28T12:47:15.851 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:15 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:15.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:15 smithi084 ceph-mon[28081]: from='client.? 172.21.15.179:0/2114104478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6babaf1e-dfe0-4f33-a484-ba66a566d5d3"}]: dispatch 2024-04-28T12:47:15.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:15 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6babaf1e-dfe0-4f33-a484-ba66a566d5d3"}]: dispatch 2024-04-28T12:47:15.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:15 smithi084 ceph-mon[28081]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6babaf1e-dfe0-4f33-a484-ba66a566d5d3"}]': finished 2024-04-28T12:47:15.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:15 smithi084 ceph-mon[28081]: osdmap e77: 12 total, 11 up, 12 in 2024-04-28T12:47:15.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:15 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:15.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:15 smithi161 ceph-mon[62387]: from='client.? 172.21.15.179:0/2114104478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6babaf1e-dfe0-4f33-a484-ba66a566d5d3"}]: dispatch 2024-04-28T12:47:15.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:15 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6babaf1e-dfe0-4f33-a484-ba66a566d5d3"}]: dispatch 2024-04-28T12:47:15.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:15 smithi161 ceph-mon[62387]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6babaf1e-dfe0-4f33-a484-ba66a566d5d3"}]': finished 2024-04-28T12:47:15.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:15 smithi161 ceph-mon[62387]: osdmap e77: 12 total, 11 up, 12 in 2024-04-28T12:47:15.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:15 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:16.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:16 smithi179 ceph-mon[63029]: pgmap v270: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-28T12:47:16.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:16 smithi179 ceph-mon[63029]: from='client.? 172.21.15.179:0/4039550385' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:47:16.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:16 smithi084 ceph-mon[28081]: pgmap v270: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-28T12:47:16.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:16 smithi084 ceph-mon[28081]: from='client.? 172.21.15.179:0/4039550385' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:47:16.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:16 smithi161 ceph-mon[62387]: pgmap v270: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-04-28T12:47:16.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:16 smithi161 ceph-mon[62387]: from='client.? 172.21.15.179:0/4039550385' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-28T12:47:18.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:18 smithi179 ceph-mon[63029]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:18.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:18 smithi084 ceph-mon[28081]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:18.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:18 smithi161 ceph-mon[62387]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:20.800 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:20 smithi179 ceph-mon[63029]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:20.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:20 smithi084 ceph-mon[28081]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:20.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:20 smithi161 ceph-mon[62387]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:22.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:22 smithi179 ceph-mon[63029]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:22.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:22 smithi084 ceph-mon[28081]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:22.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:22 smithi161 ceph-mon[62387]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:23.882 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:23 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-28T12:47:23.882 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:23 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:23.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:23 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-28T12:47:23.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:23 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:23.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:23 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-28T12:47:23.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:23 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:24.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:24 smithi179 ceph-mon[63029]: Deploying daemon osd.11 on smithi179 2024-04-28T12:47:24.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:24 smithi179 ceph-mon[63029]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:24.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:24 smithi084 ceph-mon[28081]: Deploying daemon osd.11 on smithi179 2024-04-28T12:47:24.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:24 smithi084 ceph-mon[28081]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:24.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:24 smithi161 ceph-mon[62387]: Deploying daemon osd.11 on smithi179 2024-04-28T12:47:24.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:24 smithi161 ceph-mon[62387]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:26.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:26 smithi179 ceph-mon[63029]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:26.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:26 smithi084 ceph-mon[28081]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:26.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:26 smithi161 ceph-mon[62387]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:28.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:28 smithi179 ceph-mon[63029]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:28.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:28 smithi084 ceph-mon[28081]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:28.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:28 smithi161 ceph-mon[62387]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:30.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:30.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:30.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:30.415 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:30 smithi179 ceph-mon[63029]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:30.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:30.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:30.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:30.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:30 smithi084 ceph-mon[28081]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:30.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:30.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:30.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:30.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:30 smithi161 ceph-mon[62387]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:32 smithi179 ceph-mon[63029]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:32.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:32.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:32 smithi084 ceph-mon[28081]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:32.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:32.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:32.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:32.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:32.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:32.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:32 smithi161 ceph-mon[62387]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:32.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:32.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:32.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:32.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:32.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:33.158 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 11 on host 'smithi179' 2024-04-28T12:47:33.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.162+0000 7ff91e7fc700 1 -- 172.21.15.179:0/1076503810 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ff928063860 con 0x7ff9140a3cc0 2024-04-28T12:47:33.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.164+0000 7ff92f3c4700 1 -- 172.21.15.179:0/1076503810 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff9140a3cc0 msgr2=0x7ff9140a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:33.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.164+0000 7ff92f3c4700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff9140a3cc0 0x7ff9140a6180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff918005950 tx=0x7ff9180058e0 comp rx=0 tx=0).stop 2024-04-28T12:47:33.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.164+0000 7ff92f3c4700 1 -- 172.21.15.179:0/1076503810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9281034a0 msgr2=0x7ff9281995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:33.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.164+0000 7ff92f3c4700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9281034a0 0x7ff9281995d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff92400b530 tx=0x7ff92400b840 comp rx=0 tx=0).stop 2024-04-28T12:47:33.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.164+0000 7ff92f3c4700 1 -- 172.21.15.179:0/1076503810 shutdown_connections 2024-04-28T12:47:33.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.164+0000 7ff92f3c4700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff9140a3cc0 0x7ff9140a6180 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:33.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.164+0000 7ff92f3c4700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff928105040 0x7ff9281a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:33.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.164+0000 7ff92f3c4700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9281034a0 0x7ff9281995d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:33.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.164+0000 7ff92f3c4700 1 --2- 172.21.15.179:0/1076503810 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9281046a0 0x7ff928199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:33.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.164+0000 7ff92f3c4700 1 -- 172.21.15.179:0/1076503810 >> 172.21.15.179:0/1076503810 conn(0x7ff9280fea40 msgr2=0x7ff9281079a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:33.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.165+0000 7ff92f3c4700 1 -- 172.21.15.179:0/1076503810 shutdown_connections 2024-04-28T12:47:33.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:33.165+0000 7ff92f3c4700 1 -- 172.21.15.179:0/1076503810 wait complete. 2024-04-28T12:47:33.830 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:33 smithi179 ceph-mon[63029]: from='osd.11 [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T12:47:33.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:33 smithi179 ceph-mon[63029]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T12:47:33.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:33 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:33.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:33 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:33.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:33 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:33.856 DEBUG:teuthology.orchestra.run.smithi179:osd.11> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.11.service 2024-04-28T12:47:33.858 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-28T12:47:33.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-28T12:47:33.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:33 smithi084 ceph-mon[28081]: from='osd.11 [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T12:47:33.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:33 smithi084 ceph-mon[28081]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T12:47:33.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:33.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:33.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:33.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:33 smithi161 ceph-mon[62387]: from='osd.11 [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T12:47:33.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:33 smithi161 ceph-mon[62387]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-28T12:47:33.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:33 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:33.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:33 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:33.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:33 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:34.300 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:47:34.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:34 smithi084 ceph-mon[28081]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:34.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:34 smithi084 ceph-mon[28081]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-28T12:47:34.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:34 smithi084 ceph-mon[28081]: from='osd.11 [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:47:34.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:34 smithi084 ceph-mon[28081]: osdmap e78: 12 total, 11 up, 12 in 2024-04-28T12:47:34.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:34.698 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:34 smithi084 ceph-mon[28081]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:47:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:34 smithi179 ceph-mon[63029]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:34 smithi179 ceph-mon[63029]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-28T12:47:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:34 smithi179 ceph-mon[63029]: from='osd.11 [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:47:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:34 smithi179 ceph-mon[63029]: osdmap e78: 12 total, 11 up, 12 in 2024-04-28T12:47:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:34 smithi179 ceph-mon[63029]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:47:34.916 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 12:47:34 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T12:47:34.652+0000 7f25d42c8700 -1 osd.11 0 waiting for initial osdmap 2024-04-28T12:47:34.916 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 12:47:34 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T12:47:34.666+0000 7f25d00bb700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-28T12:47:34.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:34 smithi161 ceph-mon[62387]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:34.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:34 smithi161 ceph-mon[62387]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-28T12:47:34.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:34 smithi161 ceph-mon[62387]: from='osd.11 [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:47:34.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:34 smithi161 ceph-mon[62387]: osdmap e78: 12 total, 11 up, 12 in 2024-04-28T12:47:34.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:34.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:34 smithi161 ceph-mon[62387]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-04-28T12:47:35.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:35 smithi179 ceph-mon[63029]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-04-28T12:47:35.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:35 smithi179 ceph-mon[63029]: osdmap e79: 12 total, 11 up, 12 in 2024-04-28T12:47:35.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:35 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:35.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:35 smithi179 ceph-mon[63029]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:35.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:35 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:35.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:35 smithi084 ceph-mon[28081]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-04-28T12:47:35.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:35 smithi084 ceph-mon[28081]: osdmap e79: 12 total, 11 up, 12 in 2024-04-28T12:47:35.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:35 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:35.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:35 smithi084 ceph-mon[28081]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:35.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:35 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:35.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:35 smithi161 ceph-mon[62387]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-04-28T12:47:35.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:35 smithi161 ceph-mon[62387]: osdmap e79: 12 total, 11 up, 12 in 2024-04-28T12:47:35.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:35 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:35.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:35 smithi161 ceph-mon[62387]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-28T12:47:35.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:35 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:36.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.675+0000 7f2f6d800700 1 -- 172.21.15.84:0/82470395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f681033b0 msgr2=0x7f2f681037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:36.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.675+0000 7f2f6d800700 1 --2- 172.21.15.84:0/82470395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f681033b0 0x7f2f681037d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2f580099c0 tx=0x7f2f58009cd0 comp rx=0 tx=0).stop 2024-04-28T12:47:36.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.675+0000 7f2f6d800700 1 -- 172.21.15.84:0/82470395 shutdown_connections 2024-04-28T12:47:36.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.675+0000 7f2f6d800700 1 --2- 172.21.15.84:0/82470395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f681033b0 0x7f2f681037d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:36.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.675+0000 7f2f6d800700 1 --2- 172.21.15.84:0/82470395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f68104f50 0x7f2f6810b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:36.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.675+0000 7f2f6d800700 1 --2- 172.21.15.84:0/82470395 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f681045b0 0x7f2f68104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:36.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.675+0000 7f2f6d800700 1 -- 172.21.15.84:0/82470395 >> 172.21.15.84:0/82470395 conn(0x7f2f680fe950 msgr2=0x7f2f68100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:36.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.675+0000 7f2f6d800700 1 -- 172.21.15.84:0/82470395 shutdown_connections 2024-04-28T12:47:36.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.675+0000 7f2f6d800700 1 -- 172.21.15.84:0/82470395 wait complete. 2024-04-28T12:47:36.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.677+0000 7f2f6d800700 1 Processor -- start 2024-04-28T12:47:36.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.677+0000 7f2f6d800700 1 -- start start 2024-04-28T12:47:36.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.678+0000 7f2f6d800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f681033b0 0x7f2f6819b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:36.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.678+0000 7f2f6d800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f681045b0 0x7f2f6819bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:36.674 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.678+0000 7f2f6d800700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f68104f50 0x7f2f681a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.678+0000 7f2f6d800700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2f6810ee70 con 0x7f2f681033b0 2024-04-28T12:47:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.678+0000 7f2f6d800700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2f6810ecf0 con 0x7f2f68104f50 2024-04-28T12:47:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.678+0000 7f2f6d800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2f6810eff0 con 0x7f2f681045b0 2024-04-28T12:47:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.678+0000 7f2f67e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f681033b0 0x7f2f6819b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.678+0000 7f2f67400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f681045b0 0x7f2f6819bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.678+0000 7f2f67e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f681033b0 0x7f2f6819b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52000/0 (socket says 172.21.15.84:52000) 2024-04-28T12:47:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.678+0000 7f2f67e00700 1 -- 172.21.15.84:0/3355781376 learned_addr learned my addr 172.21.15.84:0/3355781376 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:47:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.678+0000 7f2f67400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f681045b0 0x7f2f6819bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:35888/0 (socket says 172.21.15.84:35888) 2024-04-28T12:47:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.678+0000 7f2f6ce00700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f68104f50 0x7f2f681a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.679+0000 7f2f67400700 1 -- 172.21.15.84:0/3355781376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f68104f50 msgr2=0x7f2f681a2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.679+0000 7f2f67400700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f68104f50 0x7f2f681a2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.679+0000 7f2f67400700 1 -- 172.21.15.84:0/3355781376 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f681033b0 msgr2=0x7f2f6819b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:36.676 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.679+0000 7f2f67400700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f681033b0 0x7f2f6819b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:36.676 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.679+0000 7f2f67400700 1 -- 172.21.15.84:0/3355781376 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f681a3330 con 0x7f2f681045b0 2024-04-28T12:47:36.676 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.679+0000 7f2f67400700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f681045b0 0x7f2f6819bd00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400ea20 tx=0x7f2f5400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:36.676 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.680+0000 7f2f64c00700 1 -- 172.21.15.84:0/3355781376 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5400cbd0 con 0x7f2f681045b0 2024-04-28T12:47:36.676 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.680+0000 7f2f64c00700 1 -- 172.21.15.84:0/3355781376 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f5400ee60 con 0x7f2f681045b0 2024-04-28T12:47:36.676 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.680+0000 7f2f6d800700 1 -- 172.21.15.84:0/3355781376 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f681a3620 con 0x7f2f681045b0 2024-04-28T12:47:36.676 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.680+0000 7f2f64c00700 1 -- 172.21.15.84:0/3355781376 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f54018680 con 0x7f2f681045b0 2024-04-28T12:47:36.676 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.680+0000 7f2f6d800700 1 -- 172.21.15.84:0/3355781376 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f681a3bb0 con 0x7f2f681045b0 2024-04-28T12:47:36.677 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.681+0000 7f2f6d800700 1 -- 172.21.15.84:0/3355781376 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f6806a110 con 0x7f2f681045b0 2024-04-28T12:47:36.679 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.684+0000 7f2f64c00700 1 -- 172.21.15.84:0/3355781376 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2f54010430 con 0x7f2f681045b0 2024-04-28T12:47:36.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.686+0000 7f2f64c00700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2f4c0a3cc0 0x7f2f4c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:36.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.686+0000 7f2f64c00700 1 -- 172.21.15.84:0/3355781376 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2f540cfff0 con 0x7f2f681045b0 2024-04-28T12:47:36.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.686+0000 7f2f67e00700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2f4c0a3cc0 0x7f2f4c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:36.682 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.687+0000 7f2f67e00700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2f4c0a3cc0 0x7f2f4c0a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2f58003820 tx=0x7f2f58005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:36.683 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.687+0000 7f2f64c00700 1 -- 172.21.15.84:0/3355781376 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2f5409ce10 con 0x7f2f681045b0 2024-04-28T12:47:36.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.913+0000 7f2f6d800700 1 -- 172.21.15.84:0/3355781376 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f2f681a3ed0 con 0x7f2f681045b0 2024-04-28T12:47:36.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.913+0000 7f2f64c00700 1 -- 172.21.15.84:0/3355781376 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f2f5409c710 con 0x7f2f681045b0 2024-04-28T12:47:36.909 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:47:36.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.916+0000 7f2f6d800700 1 -- 172.21.15.84:0/3355781376 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2f4c0a3cc0 msgr2=0x7f2f4c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:36.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.916+0000 7f2f6d800700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2f4c0a3cc0 0x7f2f4c0a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2f58003820 tx=0x7f2f58005fb0 comp rx=0 tx=0).stop 2024-04-28T12:47:36.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.916+0000 7f2f6d800700 1 -- 172.21.15.84:0/3355781376 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f681045b0 msgr2=0x7f2f6819bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:36.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.916+0000 7f2f6d800700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f681045b0 0x7f2f6819bd00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400ea20 tx=0x7f2f5400ed30 comp rx=0 tx=0).stop 2024-04-28T12:47:36.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.916+0000 7f2f6d800700 1 -- 172.21.15.84:0/3355781376 shutdown_connections 2024-04-28T12:47:36.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.916+0000 7f2f6d800700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f2f4c0a3cc0 0x7f2f4c0a6180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:36.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.916+0000 7f2f6d800700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f68104f50 0x7f2f681a2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:36.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.916+0000 7f2f6d800700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f681033b0 0x7f2f6819b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:36.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.916+0000 7f2f6d800700 1 --2- 172.21.15.84:0/3355781376 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2f681045b0 0x7f2f6819bd00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:36.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.916+0000 7f2f6d800700 1 -- 172.21.15.84:0/3355781376 >> 172.21.15.84:0/3355781376 conn(0x7f2f680fe950 msgr2=0x7f2f68075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:36.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.917+0000 7f2f6d800700 1 -- 172.21.15.84:0/3355781376 shutdown_connections 2024-04-28T12:47:36.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:36.917+0000 7f2f6d800700 1 -- 172.21.15.84:0/3355781376 wait complete. 2024-04-28T12:47:36.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:36 smithi084 ceph-mon[28081]: purged_snaps scrub starts 2024-04-28T12:47:36.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:36 smithi084 ceph-mon[28081]: purged_snaps scrub ok 2024-04-28T12:47:36.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:36 smithi084 ceph-mon[28081]: osd.11 [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890] boot 2024-04-28T12:47:36.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:36 smithi084 ceph-mon[28081]: osdmap e80: 12 total, 12 up, 12 in 2024-04-28T12:47:36.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:36 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:36.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:36 smithi161 ceph-mon[62387]: purged_snaps scrub starts 2024-04-28T12:47:36.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:36 smithi161 ceph-mon[62387]: purged_snaps scrub ok 2024-04-28T12:47:36.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:36 smithi161 ceph-mon[62387]: osd.11 [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890] boot 2024-04-28T12:47:36.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:36 smithi161 ceph-mon[62387]: osdmap e80: 12 total, 12 up, 12 in 2024-04-28T12:47:36.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:36 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:37.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:36 smithi179 ceph-mon[63029]: purged_snaps scrub starts 2024-04-28T12:47:37.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:36 smithi179 ceph-mon[63029]: purged_snaps scrub ok 2024-04-28T12:47:37.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:36 smithi179 ceph-mon[63029]: osd.11 [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890] boot 2024-04-28T12:47:37.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:36 smithi179 ceph-mon[63029]: osdmap e80: 12 total, 12 up, 12 in 2024-04-28T12:47:37.060 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:36 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-28T12:47:37.718 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1714308455,"num_in_osds":12,"osd_in_since":1714308435,"num_remapped_pgs":0} 2024-04-28T12:47:37.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-28T12:47:37.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: osdmap e81: 12 total, 12 up, 12 in 2024-04-28T12:47:37.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3355781376' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-28T12:47:37.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: Detected new or changed devices on smithi179 2024-04-28T12:47:37.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:37.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:37.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:37.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:37.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:37.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:37.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi179 to 4272M 2024-04-28T12:47:37.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:37.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:37.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:37.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:37.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:37 smithi084 ceph-mon[28081]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:37.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: osdmap e81: 12 total, 12 up, 12 in 2024-04-28T12:47:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3355781376' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-28T12:47:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: Detected new or changed devices on smithi179 2024-04-28T12:47:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi179 to 4272M 2024-04-28T12:47:37.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:37.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:37.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:37.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:37.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:37 smithi161 ceph-mon[62387]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:38.142 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:47:38.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: osdmap e81: 12 total, 12 up, 12 in 2024-04-28T12:47:38.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3355781376' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-28T12:47:38.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: Detected new or changed devices on smithi179 2024-04-28T12:47:38.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:38.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:38.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:38.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:38.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:38.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:47:38.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi179 to 4272M 2024-04-28T12:47:38.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:38.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:38.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:38.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:38.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:37 smithi179 ceph-mon[63029]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:40.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.435+0000 7f27f6a00700 1 -- 172.21.15.84:0/4099955783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27f010b6a0 msgr2=0x7f27f010ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:40.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.435+0000 7f27f6a00700 1 --2- 172.21.15.84:0/4099955783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27f010b6a0 0x7f27f010ba80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009990 tx=0x7f27e0009ca0 comp rx=0 tx=0).stop 2024-04-28T12:47:40.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.435+0000 7f27f6a00700 1 -- 172.21.15.84:0/4099955783 shutdown_connections 2024-04-28T12:47:40.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.435+0000 7f27f6a00700 1 --2- 172.21.15.84:0/4099955783 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f0105fd0 0x7f27f01124e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:40.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.435+0000 7f27f6a00700 1 --2- 172.21.15.84:0/4099955783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27f010b6a0 0x7f27f010ba80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:40.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.435+0000 7f27f6a00700 1 --2- 172.21.15.84:0/4099955783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f0105610 0x7f27f0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:40.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.435+0000 7f27f6a00700 1 -- 172.21.15.84:0/4099955783 >> 172.21.15.84:0/4099955783 conn(0x7f27f0101100 msgr2=0x7f27f0103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:40.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.436+0000 7f27f6a00700 1 -- 172.21.15.84:0/4099955783 shutdown_connections 2024-04-28T12:47:40.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.436+0000 7f27f6a00700 1 -- 172.21.15.84:0/4099955783 wait complete. 2024-04-28T12:47:40.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.437+0000 7f27f6a00700 1 Processor -- start 2024-04-28T12:47:40.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.437+0000 7f27f6a00700 1 -- start start 2024-04-28T12:47:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.438+0000 7f27f6a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27f0105610 0x7f27f01a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.438+0000 7f27f6a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f0105fd0 0x7f27f01a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.438+0000 7f27f6a00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f010b6a0 0x7f27f019bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.438+0000 7f27f6a00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0115860 con 0x7f27f0105610 2024-04-28T12:47:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.438+0000 7f27f6a00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f27f01156e0 con 0x7f27f010b6a0 2024-04-28T12:47:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.438+0000 7f27f6a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f27f01159e0 con 0x7f27f0105fd0 2024-04-28T12:47:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.438+0000 7f27efe00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27f0105610 0x7f27f01a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.438+0000 7f27efe00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27f0105610 0x7f27f01a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59290/0 (socket says 172.21.15.84:59290) 2024-04-28T12:47:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.438+0000 7f27efe00700 1 -- 172.21.15.84:0/2383422275 learned_addr learned my addr 172.21.15.84:0/2383422275 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:47:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.438+0000 7f27e7e00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f0105fd0 0x7f27f01a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.438+0000 7f27f4a00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f010b6a0 0x7f27f019bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:40.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.439+0000 7f27efe00700 1 -- 172.21.15.84:0/2383422275 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f0105fd0 msgr2=0x7f27f01a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:40.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.439+0000 7f27efe00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f0105fd0 0x7f27f01a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:40.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.439+0000 7f27efe00700 1 -- 172.21.15.84:0/2383422275 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f010b6a0 msgr2=0x7f27f019bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:40.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.439+0000 7f27efe00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f010b6a0 0x7f27f019bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:40.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.439+0000 7f27efe00700 1 -- 172.21.15.84:0/2383422275 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f019c240 con 0x7f27f0105610 2024-04-28T12:47:40.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.439+0000 7f27e7e00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f0105fd0 0x7f27f01a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:47:40.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.439+0000 7f27efe00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27f0105610 0x7f27f01a1920 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f27e0000c00 tx=0x7f27e00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:40.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.439+0000 7f27ed600700 1 -- 172.21.15.84:0/2383422275 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0027070 con 0x7f27f0105610 2024-04-28T12:47:40.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.439+0000 7f27ed600700 1 -- 172.21.15.84:0/2383422275 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27e0022470 con 0x7f27f0105610 2024-04-28T12:47:40.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.439+0000 7f27ed600700 1 -- 172.21.15.84:0/2383422275 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0017650 con 0x7f27f0105610 2024-04-28T12:47:40.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.439+0000 7f27f6a00700 1 -- 172.21.15.84:0/2383422275 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f019c4d0 con 0x7f27f0105610 2024-04-28T12:47:40.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.439+0000 7f27f6a00700 1 -- 172.21.15.84:0/2383422275 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f01a87f0 con 0x7f27f0105610 2024-04-28T12:47:40.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.441+0000 7f27f6a00700 1 -- 172.21.15.84:0/2383422275 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f006a110 con 0x7f27f0105610 2024-04-28T12:47:40.438 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.442+0000 7f27ed600700 1 -- 172.21.15.84:0/2383422275 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f27e002a070 con 0x7f27f0105610 2024-04-28T12:47:40.439 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.444+0000 7f27ed600700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f27d40a3cc0 0x7f27d40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:40.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.444+0000 7f27ed600700 1 -- 172.21.15.84:0/2383422275 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f27e00d0b80 con 0x7f27f0105610 2024-04-28T12:47:40.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.444+0000 7f27e7e00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f27d40a3cc0 0x7f27d40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:40.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.445+0000 7f27e7e00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f27d40a3cc0 0x7f27d40a6180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f27f010ada0 tx=0x7f27d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:40.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.449+0000 7f27ed600700 1 -- 172.21.15.84:0/2383422275 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f27e0013070 con 0x7f27f0105610 2024-04-28T12:47:40.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.677+0000 7f27f6a00700 1 -- 172.21.15.84:0/2383422275 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f27f002e490 con 0x7f27f0105610 2024-04-28T12:47:40.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.679+0000 7f27ed600700 1 -- 172.21.15.84:0/2383422275 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18445 (secure 0 0 0) 0x7f27e009d9f0 con 0x7f27f0105610 2024-04-28T12:47:40.675 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:47:40.676 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":81,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:47:36.668828+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:47:40.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.682+0000 7f27f6a00700 1 -- 172.21.15.84:0/2383422275 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f27d40a3cc0 msgr2=0x7f27d40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:40.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.682+0000 7f27f6a00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f27d40a3cc0 0x7f27d40a6180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f27f010ada0 tx=0x7f27d8009260 comp rx=0 tx=0).stop 2024-04-28T12:47:40.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.682+0000 7f27f6a00700 1 -- 172.21.15.84:0/2383422275 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27f0105610 msgr2=0x7f27f01a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:40.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.682+0000 7f27f6a00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27f0105610 0x7f27f01a1920 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f27e0000c00 tx=0x7f27e00057c0 comp rx=0 tx=0).stop 2024-04-28T12:47:40.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.682+0000 7f27f6a00700 1 -- 172.21.15.84:0/2383422275 shutdown_connections 2024-04-28T12:47:40.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.682+0000 7f27f6a00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f27d40a3cc0 0x7f27d40a6180 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:40.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.682+0000 7f27f6a00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f010b6a0 0x7f27f019bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:40.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.682+0000 7f27f6a00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27f0105610 0x7f27f01a1920 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:40.678 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.682+0000 7f27f6a00700 1 --2- 172.21.15.84:0/2383422275 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f0105fd0 0x7f27f01a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:40.679 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.683+0000 7f27f6a00700 1 -- 172.21.15.84:0/2383422275 >> 172.21.15.84:0/2383422275 conn(0x7f27f0101100 msgr2=0x7f27f0102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:40.679 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.683+0000 7f27f6a00700 1 -- 172.21.15.84:0/2383422275 shutdown_connections 2024-04-28T12:47:40.679 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:40.683+0000 7f27f6a00700 1 -- 172.21.15.84:0/2383422275 wait complete. 2024-04-28T12:47:40.696 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:40 smithi084 ceph-mon[28081]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:40 smithi179 ceph-mon[63029]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:40.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:40 smithi161 ceph-mon[62387]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:41.510 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:43:15.608080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T12:47:41.511 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-28T12:47:41.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:41 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2383422275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:47:41.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:41 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2383422275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:47:41.947 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:47:41.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:41 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2383422275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:47:42.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:42 smithi179 ceph-mon[63029]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:42.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:42 smithi084 ceph-mon[28081]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:42.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:42 smithi161 ceph-mon[62387]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:44.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.254+0000 7f6cbb200700 1 -- 172.21.15.84:0/2417460683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc102120 msgr2=0x7f6cbc102500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:44.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.254+0000 7f6cbb200700 1 --2- 172.21.15.84:0/2417460683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc102120 0x7f6cbc102500 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6cac009990 tx=0x7f6cac009ca0 comp rx=0 tx=0).stop 2024-04-28T12:47:44.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.254+0000 7f6cbb200700 1 -- 172.21.15.84:0/2417460683 shutdown_connections 2024-04-28T12:47:44.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.254+0000 7f6cbb200700 1 --2- 172.21.15.84:0/2417460683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc102120 0x7f6cbc102500 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:44.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.255+0000 7f6cbb200700 1 --2- 172.21.15.84:0/2417460683 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cbc10fcd0 0x7f6cbc1120c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:44.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.255+0000 7f6cbb200700 1 --2- 172.21.15.84:0/2417460683 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cbc102ad0 0x7f6cbc10f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:44.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.255+0000 7f6cbb200700 1 -- 172.21.15.84:0/2417460683 >> 172.21.15.84:0/2417460683 conn(0x7f6cbc0fdd20 msgr2=0x7f6cbc100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:44.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.255+0000 7f6cbb200700 1 -- 172.21.15.84:0/2417460683 shutdown_connections 2024-04-28T12:47:44.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.255+0000 7f6cbb200700 1 -- 172.21.15.84:0/2417460683 wait complete. 2024-04-28T12:47:44.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.256+0000 7f6cbb200700 1 Processor -- start 2024-04-28T12:47:44.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.256+0000 7f6cbb200700 1 -- start start 2024-04-28T12:47:44.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.256+0000 7f6cbb200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cbc102120 0x7f6cbc1a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:44.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.256+0000 7f6cbb200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cbc102ad0 0x7f6cbc1a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:44.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.256+0000 7f6cbb200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc10fcd0 0x7f6cbc19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:44.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.256+0000 7f6cbb200700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc1157b0 con 0x7f6cbc102ad0 2024-04-28T12:47:44.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.256+0000 7f6cbb200700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc115630 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.256+0000 7f6cbb200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc115930 con 0x7f6cbc102120 2024-04-28T12:47:44.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.257+0000 7f6cb9400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cbc102ad0 0x7f6cbc1a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.257+0000 7f6cba800700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc10fcd0 0x7f6cbc19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.257+0000 7f6cb9400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cbc102ad0 0x7f6cbc1a1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59304/0 (socket says 172.21.15.84:59304) 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.257+0000 7f6cb9400700 1 -- 172.21.15.84:0/2062403659 learned_addr learned my addr 172.21.15.84:0/2062403659 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.257+0000 7f6cb9e00700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cbc102120 0x7f6cbc1a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.257+0000 7f6cba800700 1 -- 172.21.15.84:0/2062403659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cbc102120 msgr2=0x7f6cbc1a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.257+0000 7f6cba800700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cbc102120 0x7f6cbc1a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.257+0000 7f6cba800700 1 -- 172.21.15.84:0/2062403659 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cbc102ad0 msgr2=0x7f6cbc1a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.257+0000 7f6cba800700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cbc102ad0 0x7f6cbc1a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.257+0000 7f6cba800700 1 -- 172.21.15.84:0/2062403659 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc19c230 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.257+0000 7f6cb9400700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cbc102ad0 0x7f6cbc1a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.257+0000 7f6cba800700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc10fcd0 0x7f6cbc19bad0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc06ab00 tx=0x7f6cb400ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.258+0000 7f6cb2a00700 1 -- 172.21.15.84:0/2062403659 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb40092e0 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.258+0000 7f6cb2a00700 1 -- 172.21.15.84:0/2062403659 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cb4004bb0 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.258+0000 7f6cb2a00700 1 -- 172.21.15.84:0/2062403659 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4004020 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.258+0000 7f6cbb200700 1 -- 172.21.15.84:0/2062403659 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cbc19c4c0 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.258+0000 7f6cbb200700 1 -- 172.21.15.84:0/2062403659 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cbc1a87e0 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.260+0000 7f6cbb200700 1 -- 172.21.15.84:0/2062403659 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cbc06a110 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.257 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.261+0000 7f6cb2a00700 1 -- 172.21.15.84:0/2062403659 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6cb401b030 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.263+0000 7f6cb2a00700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6ca00a3cc0 0x7f6ca00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:44.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.263+0000 7f6cb2a00700 1 -- 172.21.15.84:0/2062403659 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6cb40cf970 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.263+0000 7f6cb9e00700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6ca00a3cc0 0x7f6ca00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:44.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.264+0000 7f6cb9e00700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6ca00a3cc0 0x7f6ca00a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6cac00b2e0 tx=0x7f6cac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:44.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.267+0000 7f6cb2a00700 1 -- 172.21.15.84:0/2062403659 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6cb4022020 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.487 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.491+0000 7f6cbb200700 1 -- 172.21.15.84:0/2062403659 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6cbc02e490 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.492+0000 7f6cb2a00700 1 -- 172.21.15.84:0/2062403659 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f6cb4010040 con 0x7f6cbc10fcd0 2024-04-28T12:47:44.488 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-04-28T12:47:44.492 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.497+0000 7f6cbb200700 1 -- 172.21.15.84:0/2062403659 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6ca00a3cc0 msgr2=0x7f6ca00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:44.492 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.497+0000 7f6cbb200700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6ca00a3cc0 0x7f6ca00a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6cac00b2e0 tx=0x7f6cac005fd0 comp rx=0 tx=0).stop 2024-04-28T12:47:44.493 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.497+0000 7f6cbb200700 1 -- 172.21.15.84:0/2062403659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc10fcd0 msgr2=0x7f6cbc19bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:44.493 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.497+0000 7f6cbb200700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc10fcd0 0x7f6cbc19bad0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc06ab00 tx=0x7f6cb400ef90 comp rx=0 tx=0).stop 2024-04-28T12:47:44.493 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.497+0000 7f6cbb200700 1 -- 172.21.15.84:0/2062403659 shutdown_connections 2024-04-28T12:47:44.493 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.497+0000 7f6cbb200700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6ca00a3cc0 0x7f6ca00a6180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:44.493 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.497+0000 7f6cbb200700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc10fcd0 0x7f6cbc19bad0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:44.493 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.497+0000 7f6cbb200700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6cbc102120 0x7f6cbc1a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:44.493 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.497+0000 7f6cbb200700 1 --2- 172.21.15.84:0/2062403659 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cbc102ad0 0x7f6cbc1a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:44.493 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.497+0000 7f6cbb200700 1 -- 172.21.15.84:0/2062403659 >> 172.21.15.84:0/2062403659 conn(0x7f6cbc0fdd20 msgr2=0x7f6cbc100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:44.493 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.497+0000 7f6cbb200700 1 -- 172.21.15.84:0/2062403659 shutdown_connections 2024-04-28T12:47:44.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:44.498+0000 7f6cbb200700 1 -- 172.21.15.84:0/2062403659 wait complete. 2024-04-28T12:47:44.836 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:44 smithi084 ceph-mon[28081]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:44.836 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:44 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2062403659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:47:44.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:44 smithi179 ceph-mon[63029]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:44.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:44 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2062403659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:47:44.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:44 smithi161 ceph-mon[62387]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:44.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:44 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2062403659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:47:45.263 INFO:tasks.cephadm:Adding mds.a on smithi084 2024-04-28T12:47:45.263 INFO:tasks.cephadm:Adding mds.d on smithi084 2024-04-28T12:47:45.263 INFO:tasks.cephadm:Adding mds.g on smithi084 2024-04-28T12:47:45.263 INFO:tasks.cephadm:Adding mds.j on smithi084 2024-04-28T12:47:45.263 INFO:tasks.cephadm:Adding mds.b on smithi161 2024-04-28T12:47:45.263 INFO:tasks.cephadm:Adding mds.e on smithi161 2024-04-28T12:47:45.264 INFO:tasks.cephadm:Adding mds.h on smithi161 2024-04-28T12:47:45.264 INFO:tasks.cephadm:Adding mds.k on smithi161 2024-04-28T12:47:45.264 INFO:tasks.cephadm:Adding mds.c on smithi179 2024-04-28T12:47:45.264 INFO:tasks.cephadm:Adding mds.f on smithi179 2024-04-28T12:47:45.264 INFO:tasks.cephadm:Adding mds.i on smithi179 2024-04-28T12:47:45.264 INFO:tasks.cephadm:Adding mds.l on smithi179 2024-04-28T12:47:45.264 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi084=a;smithi084=d;smithi084=g;smithi084=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l' 2024-04-28T12:47:45.513 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.c/config 2024-04-28T12:47:46.778 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:46 smithi179 ceph-mon[63029]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:46.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:46 smithi084 ceph-mon[28081]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:46.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:46 smithi161 ceph-mon[62387]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:47.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.315+0000 7f697179c700 1 -- 172.21.15.179:0/4095578943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c104810 msgr2=0x7f696c104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:47.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.315+0000 7f697179c700 1 --2- 172.21.15.179:0/4095578943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c104810 0x7f696c104c70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6960009990 tx=0x7f6960009ca0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.315+0000 7f697179c700 1 -- 172.21.15.179:0/4095578943 shutdown_connections 2024-04-28T12:47:47.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.315+0000 7f697179c700 1 --2- 172.21.15.179:0/4095578943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c101790 0x7f696c101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.315+0000 7f697179c700 1 --2- 172.21.15.179:0/4095578943 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f696c1051b0 0x7f696c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.315+0000 7f697179c700 1 --2- 172.21.15.179:0/4095578943 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c104810 0x7f696c104c70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.315+0000 7f697179c700 1 -- 172.21.15.179:0/4095578943 >> 172.21.15.179:0/4095578943 conn(0x7f696c0fd0d0 msgr2=0x7f696c0ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.315+0000 7f697179c700 1 -- 172.21.15.179:0/4095578943 shutdown_connections 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.315+0000 7f697179c700 1 -- 172.21.15.179:0/4095578943 wait complete. 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.315+0000 7f697179c700 1 Processor -- start 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f697179c700 1 -- start start 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f697179c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c101790 0x7f696c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f697179c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f696c104810 0x7f696c199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f697179c700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c1051b0 0x7f696c1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f697179c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f696c10efa0 con 0x7f696c104810 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f697179c700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f696c10ee20 con 0x7f696c101790 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f697179c700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f696c10f120 con 0x7f696c1051b0 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f696b7fe700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c1051b0 0x7f696c1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f696b7fe700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c1051b0 0x7f696c1a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:47318/0 (socket says 172.21.15.179:47318) 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f696b7fe700 1 -- 172.21.15.179:0/2587980792 learned_addr learned my addr 172.21.15.179:0/2587980792 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-04-28T12:47:47.325 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f696affd700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c101790 0x7f696c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f696b7fe700 1 -- 172.21.15.179:0/2587980792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c101790 msgr2=0x7f696c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f696b7fe700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c101790 0x7f696c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f696b7fe700 1 -- 172.21.15.179:0/2587980792 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f696c104810 msgr2=0x7f696c199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f696b7fe700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f696c104810 0x7f696c199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f696b7fe700 1 -- 172.21.15.179:0/2587980792 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696c1a1240 con 0x7f696c1051b0 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f696b7fe700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c1051b0 0x7f696c1a0ba0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f696400b9a0 tx=0x7f696400bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f6953fff700 1 -- 172.21.15.179:0/2587980792 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6964019070 con 0x7f696c1051b0 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f697179c700 1 -- 172.21.15.179:0/2587980792 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f696c1a1530 con 0x7f696c1051b0 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f6953fff700 1 -- 172.21.15.179:0/2587980792 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6964014440 con 0x7f696c1051b0 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.316+0000 7f6953fff700 1 -- 172.21.15.179:0/2587980792 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f696400ce60 con 0x7f696c1051b0 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.317+0000 7f697179c700 1 -- 172.21.15.179:0/2587980792 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f696c1a1a20 con 0x7f696c1051b0 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.318+0000 7f697179c700 1 -- 172.21.15.179:0/2587980792 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f696c06a110 con 0x7f696c1051b0 2024-04-28T12:47:47.326 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.319+0000 7f6953fff700 1 -- 172.21.15.179:0/2587980792 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f696401c070 con 0x7f696c1051b0 2024-04-28T12:47:47.327 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.319+0000 7f6953fff700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69540a3c40 0x7f69540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:47.327 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.320+0000 7f6953fff700 1 -- 172.21.15.179:0/2587980792 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f69640cf3a0 con 0x7f696c1051b0 2024-04-28T12:47:47.327 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.320+0000 7f696affd700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69540a3c40 0x7f69540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:47.327 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.321+0000 7f696affd700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69540a3c40 0x7f69540a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f695c005950 tx=0x7f695c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:47.327 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.324+0000 7f6953fff700 1 -- 172.21.15.179:0/2587980792 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f696409c2b0 con 0x7f696c1051b0 2024-04-28T12:47:47.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.534+0000 7f697179c700 1 -- 172.21.15.179:0/2587980792 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi084=a;smithi084=d;smithi084=g;smithi084=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l", "target": ["mon-mgr", ""]}) -- 0x7f696c063a60 con 0x7f69540a3c40 2024-04-28T12:47:47.547 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.551+0000 7f6953fff700 1 -- 172.21.15.179:0/2587980792 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f696c063a60 con 0x7f69540a3c40 2024-04-28T12:47:47.547 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mds.all update... 2024-04-28T12:47:47.549 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.553+0000 7f697179c700 1 -- 172.21.15.179:0/2587980792 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69540a3c40 msgr2=0x7f69540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:47.549 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.553+0000 7f697179c700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69540a3c40 0x7f69540a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f695c005950 tx=0x7f695c0058e0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.549 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.553+0000 7f697179c700 1 -- 172.21.15.179:0/2587980792 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c1051b0 msgr2=0x7f696c1a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:47.549 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.553+0000 7f697179c700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c1051b0 0x7f696c1a0ba0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f696400b9a0 tx=0x7f696400bd60 comp rx=0 tx=0).stop 2024-04-28T12:47:47.550 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.553+0000 7f697179c700 1 -- 172.21.15.179:0/2587980792 shutdown_connections 2024-04-28T12:47:47.550 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.553+0000 7f697179c700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69540a3c40 0x7f69540a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.550 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.553+0000 7f697179c700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c101790 0x7f696c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.550 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.553+0000 7f697179c700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f696c1051b0 0x7f696c1a0ba0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.550 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.553+0000 7f697179c700 1 --2- 172.21.15.179:0/2587980792 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f696c104810 0x7f696c199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:47.550 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.553+0000 7f697179c700 1 -- 172.21.15.179:0/2587980792 >> 172.21.15.179:0/2587980792 conn(0x7f696c0fd0d0 msgr2=0x7f696c107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:47.550 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.553+0000 7f697179c700 1 -- 172.21.15.179:0/2587980792 shutdown_connections 2024-04-28T12:47:47.550 INFO:teuthology.orchestra.run.smithi179.stderr:2024-04-28T12:47:47.553+0000 7f697179c700 1 -- 172.21.15.179:0/2587980792 wait complete. 2024-04-28T12:47:48.354 DEBUG:teuthology.orchestra.run.smithi084:mds.a> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.a.service 2024-04-28T12:47:48.362 DEBUG:teuthology.orchestra.run.smithi084:mds.d> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.d.service 2024-04-28T12:47:48.407 DEBUG:teuthology.orchestra.run.smithi084:mds.g> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.g.service 2024-04-28T12:47:48.452 DEBUG:teuthology.orchestra.run.smithi084:mds.j> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.j.service 2024-04-28T12:47:48.495 DEBUG:teuthology.orchestra.run.smithi161:mds.b> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.b.service 2024-04-28T12:47:48.498 DEBUG:teuthology.orchestra.run.smithi161:mds.e> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.e.service 2024-04-28T12:47:48.542 DEBUG:teuthology.orchestra.run.smithi161:mds.h> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.h.service 2024-04-28T12:47:48.586 DEBUG:teuthology.orchestra.run.smithi161:mds.k> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.k.service 2024-04-28T12:47:48.619 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:48 smithi179 ceph-mon[63029]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi084=a;smithi084=d;smithi084=g;smithi084=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:47:48.620 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:48 smithi179 ceph-mon[63029]: Saving service mds.all spec with placement smithi084=a;smithi084=d;smithi084=g;smithi084=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l;count:12 2024-04-28T12:47:48.620 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:48 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:48.620 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:48 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:48.620 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:48 smithi179 ceph-mon[63029]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:48.630 DEBUG:teuthology.orchestra.run.smithi179:mds.c> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.c.service 2024-04-28T12:47:48.633 DEBUG:teuthology.orchestra.run.smithi179:mds.f> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.f.service 2024-04-28T12:47:48.677 DEBUG:teuthology.orchestra.run.smithi179:mds.i> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.i.service 2024-04-28T12:47:48.721 DEBUG:teuthology.orchestra.run.smithi179:mds.l> sudo journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.l.service 2024-04-28T12:47:48.764 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-28T12:47:48.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:47:48.843 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:48 smithi084 ceph-mon[28081]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi084=a;smithi084=d;smithi084=g;smithi084=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:47:48.843 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:48 smithi084 ceph-mon[28081]: Saving service mds.all spec with placement smithi084=a;smithi084=d;smithi084=g;smithi084=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l;count:12 2024-04-28T12:47:48.843 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:48 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:48.843 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:48 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:48.843 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:48 smithi084 ceph-mon[28081]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:48.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:48 smithi161 ceph-mon[62387]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi084=a;smithi084=d;smithi084=g;smithi084=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:47:48.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:48 smithi161 ceph-mon[62387]: Saving service mds.all spec with placement smithi084=a;smithi084=d;smithi084=g;smithi084=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l;count:12 2024-04-28T12:47:48.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:48 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:48.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:48 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:47:48.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:48 smithi161 ceph-mon[62387]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:48.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.000+0000 7f1933200640 1 -- 172.21.15.84:0/1675658156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1934108d90 msgr2=0x7f19341091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:48.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.000+0000 7f1933200640 1 --2- 172.21.15.84:0/1675658156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1934108d90 0x7f19341091b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1924009900 tx=0x7f192402a3f0 comp rx=0 tx=0).stop 2024-04-28T12:47:48.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.000+0000 7f1933200640 1 -- 172.21.15.84:0/1675658156 shutdown_connections 2024-04-28T12:47:48.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.000+0000 7f1933200640 1 --2- 172.21.15.84:0/1675658156 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f193410a940 0x7f19341111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:48.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.000+0000 7f1933200640 1 --2- 172.21.15.84:0/1675658156 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1934109f80 0x7f193410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:48.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.000+0000 7f1933200640 1 --2- 172.21.15.84:0/1675658156 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1934108d90 0x7f19341091b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:48.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.000+0000 7f1933200640 1 -- 172.21.15.84:0/1675658156 >> 172.21.15.84:0/1675658156 conn(0x7f1934077d30 msgr2=0x7f193407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:48.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.000+0000 7f1933200640 1 -- 172.21.15.84:0/1675658156 shutdown_connections 2024-04-28T12:47:48.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.000+0000 7f1933200640 1 -- 172.21.15.84:0/1675658156 wait complete. 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.001+0000 7f1933200640 1 Processor -- start 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.001+0000 7f1933200640 1 -- start start 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.001+0000 7f1933200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1934108d90 0x7f193419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1933200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1934109f80 0x7f193419f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1933200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f193410a940 0x7f19341a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1933200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f19341148b0 con 0x7f1934108d90 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1933200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1934114730 con 0x7f193410a940 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1933200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1934114a30 con 0x7f1934109f80 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1931400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1934109f80 0x7f193419f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1931400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1934109f80 0x7f193419f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:39596/0 (socket says 172.21.15.84:39596) 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1931400640 1 -- 172.21.15.84:0/1210918782 learned_addr learned my addr 172.21.15.84:0/1210918782 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1932800640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f193410a940 0x7f19341a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1931e00640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1934108d90 0x7f193419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:49.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1931400640 1 -- 172.21.15.84:0/1210918782 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f193410a940 msgr2=0x7f19341a65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:49.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1931400640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f193410a940 0x7f19341a65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1931400640 1 -- 172.21.15.84:0/1210918782 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1934108d90 msgr2=0x7f193419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:49.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1931400640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1934108d90 0x7f193419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.002+0000 7f1931400640 1 -- 172.21.15.84:0/1210918782 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19341a6c60 con 0x7f1934109f80 2024-04-28T12:47:49.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.003+0000 7f1931400640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1934109f80 0x7f193419f580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f192000e9f0 tx=0x7f192000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:49.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.003+0000 7f192aa00640 1 -- 172.21.15.84:0/1210918782 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192000ccc0 con 0x7f1934109f80 2024-04-28T12:47:49.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.003+0000 7f192aa00640 1 -- 172.21.15.84:0/1210918782 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f192000ce60 con 0x7f1934109f80 2024-04-28T12:47:49.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.003+0000 7f1933200640 1 -- 172.21.15.84:0/1210918782 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19341a6f50 con 0x7f1934109f80 2024-04-28T12:47:49.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.003+0000 7f192aa00640 1 -- 172.21.15.84:0/1210918782 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920010430 con 0x7f1934109f80 2024-04-28T12:47:49.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.003+0000 7f1933200640 1 -- 172.21.15.84:0/1210918782 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19341a73c0 con 0x7f1934109f80 2024-04-28T12:47:49.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.004+0000 7f1933200640 1 -- 172.21.15.84:0/1210918782 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1934110170 con 0x7f1934109f80 2024-04-28T12:47:49.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.006+0000 7f192aa00640 1 -- 172.21.15.84:0/1210918782 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f19200046c0 con 0x7f1934109f80 2024-04-28T12:47:49.004 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.008+0000 7f192aa00640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f19180a3bb0 0x7f19180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:49.004 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.008+0000 7f192aa00640 1 -- 172.21.15.84:0/1210918782 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f19200d0650 con 0x7f1934109f80 2024-04-28T12:47:49.004 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.008+0000 7f1931e00640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f19180a3bb0 0x7f19180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:49.005 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.008+0000 7f1931e00640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f19180a3bb0 0x7f19180a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1924000c00 tx=0x7f19240314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:49.010 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.014+0000 7f192aa00640 1 -- 172.21.15.84:0/1210918782 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1920054060 con 0x7f1934109f80 2024-04-28T12:47:49.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.217+0000 7f1933200640 1 -- 172.21.15.84:0/1210918782 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f193402dca0 con 0x7f1934109f80 2024-04-28T12:47:49.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.219+0000 7f192aa00640 1 -- 172.21.15.84:0/1210918782 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18445 (secure 0 0 0) 0x7f192009d430 con 0x7f1934109f80 2024-04-28T12:47:49.216 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:47:49.216 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":81,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:47:36.668828+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:47:49.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.222+0000 7f1933200640 1 -- 172.21.15.84:0/1210918782 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f19180a3bb0 msgr2=0x7f19180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:49.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.222+0000 7f1933200640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f19180a3bb0 0x7f19180a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1924000c00 tx=0x7f19240314a0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.222+0000 7f1933200640 1 -- 172.21.15.84:0/1210918782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1934109f80 msgr2=0x7f193419f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:49.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.222+0000 7f1933200640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1934109f80 0x7f193419f580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f192000e9f0 tx=0x7f192000edf0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.219 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.222+0000 7f1933200640 1 -- 172.21.15.84:0/1210918782 shutdown_connections 2024-04-28T12:47:49.219 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.222+0000 7f1933200640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f19180a3bb0 0x7f19180a6070 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.219 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.222+0000 7f1933200640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f193410a940 0x7f19341a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.219 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.222+0000 7f1933200640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1934109f80 0x7f193419f580 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.219 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.222+0000 7f1933200640 1 --2- 172.21.15.84:0/1210918782 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1934108d90 0x7f193419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.219 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.222+0000 7f1933200640 1 -- 172.21.15.84:0/1210918782 >> 172.21.15.84:0/1210918782 conn(0x7f1934077d30 msgr2=0x7f193410d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:49.219 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.222+0000 7f1933200640 1 -- 172.21.15.84:0/1210918782 shutdown_connections 2024-04-28T12:47:49.219 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.222+0000 7f1933200640 1 -- 172.21.15.84:0/1210918782 wait complete. 2024-04-28T12:47:49.238 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:43:15.608080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T12:47:49.239 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:47:49.438 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.441+0000 7fb9a0a00640 1 -- 172.21.15.84:0/4055686469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb99c109fc0 msgr2=0x7fb99c10a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:49.439 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.441+0000 7fb9a0a00640 1 --2- 172.21.15.84:0/4055686469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb99c109fc0 0x7fb99c10a440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb988009900 tx=0x7fb98802a410 comp rx=0 tx=0).stop 2024-04-28T12:47:49.439 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.441+0000 7fb9a0a00640 1 -- 172.21.15.84:0/4055686469 shutdown_connections 2024-04-28T12:47:49.439 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.441+0000 7fb9a0a00640 1 --2- 172.21.15.84:0/4055686469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb99c10a980 0x7fb99c1112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.439 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.441+0000 7fb9a0a00640 1 --2- 172.21.15.84:0/4055686469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb99c109fc0 0x7fb99c10a440 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.439 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.441+0000 7fb9a0a00640 1 --2- 172.21.15.84:0/4055686469 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb99c108e70 0x7fb99c109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.439 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.441+0000 7fb9a0a00640 1 -- 172.21.15.84:0/4055686469 >> 172.21.15.84:0/4055686469 conn(0x7fb99c077e00 msgr2=0x7fb99c07a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:49.439 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.441+0000 7fb9a0a00640 1 -- 172.21.15.84:0/4055686469 shutdown_connections 2024-04-28T12:47:49.439 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.441+0000 7fb9a0a00640 1 -- 172.21.15.84:0/4055686469 wait complete. 2024-04-28T12:47:49.439 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.442+0000 7fb9a0a00640 1 Processor -- start 2024-04-28T12:47:49.439 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.442+0000 7fb9a0a00640 1 -- start start 2024-04-28T12:47:49.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb9a0a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb99c108e70 0x7fb99c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:49.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb9a0a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb99c109fc0 0x7fb99c19f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:49.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb9a0a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb99c10a980 0x7fb99c1a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:49.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb9a0a00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c1149d0 con 0x7fb99c108e70 2024-04-28T12:47:49.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb9a0a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c114850 con 0x7fb99c10a980 2024-04-28T12:47:49.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb9a0a00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c114b50 con 0x7fb99c109fc0 2024-04-28T12:47:49.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb99a800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb99c108e70 0x7fb99c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:49.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb99a800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb99c108e70 0x7fb99c19f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59096/0 (socket says 172.21.15.84:59096) 2024-04-28T12:47:49.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb99a800640 1 -- 172.21.15.84:0/3858387762 learned_addr learned my addr 172.21.15.84:0/3858387762 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:47:49.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb999e00640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb99c109fc0 0x7fb99c19f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:49.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb99b200640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb99c10a980 0x7fb99c1a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:49.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb99a800640 1 -- 172.21.15.84:0/3858387762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb99c109fc0 msgr2=0x7fb99c19f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:49.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb99a800640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb99c109fc0 0x7fb99c19f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb99a800640 1 -- 172.21.15.84:0/3858387762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb99c10a980 msgr2=0x7fb99c1a6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:49.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb99a800640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb99c10a980 0x7fb99c1a6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb99a800640 1 -- 172.21.15.84:0/3858387762 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb99c1a6c90 con 0x7fb99c108e70 2024-04-28T12:47:49.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb999e00640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb99c109fc0 0x7fb99c19f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:47:49.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.443+0000 7fb99b200640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb99c10a980 0x7fb99c1a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:47:49.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.444+0000 7fb99a800640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb99c108e70 0x7fb99c19f070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00e9f0 tx=0x7fb98c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:49.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.444+0000 7fb993400640 1 -- 172.21.15.84:0/3858387762 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c00cd80 con 0x7fb99c108e70 2024-04-28T12:47:49.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.444+0000 7fb9a0a00640 1 -- 172.21.15.84:0/3858387762 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb99c1a6f80 con 0x7fb99c108e70 2024-04-28T12:47:49.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.444+0000 7fb993400640 1 -- 172.21.15.84:0/3858387762 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb98c0108e0 con 0x7fb99c108e70 2024-04-28T12:47:49.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.444+0000 7fb993400640 1 -- 172.21.15.84:0/3858387762 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c018980 con 0x7fb99c108e70 2024-04-28T12:47:49.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.444+0000 7fb9a0a00640 1 -- 172.21.15.84:0/3858387762 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb99c1a73f0 con 0x7fb99c108e70 2024-04-28T12:47:49.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.445+0000 7fb9a0a00640 1 -- 172.21.15.84:0/3858387762 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb99c110200 con 0x7fb99c108e70 2024-04-28T12:47:49.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.447+0000 7fb993400640 1 -- 172.21.15.84:0/3858387762 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb98c010430 con 0x7fb99c108e70 2024-04-28T12:47:49.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.448+0000 7fb993400640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb9800a3bb0 0x7fb9800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:49.447 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.449+0000 7fb993400640 1 -- 172.21.15.84:0/3858387762 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb98c0d0360 con 0x7fb99c108e70 2024-04-28T12:47:49.447 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.449+0000 7fb999e00640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb9800a3bb0 0x7fb9800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:49.447 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.450+0000 7fb999e00640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb9800a3bb0 0x7fb9800a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb988009230 tx=0x7fb9880314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:49.449 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.453+0000 7fb993400640 1 -- 172.21.15.84:0/3858387762 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb98c09d140 con 0x7fb99c108e70 2024-04-28T12:47:49.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.657+0000 7fb9a0a00640 1 -- 172.21.15.84:0/3858387762 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb99c02dad0 con 0x7fb99c108e70 2024-04-28T12:47:49.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.658+0000 7fb993400640 1 -- 172.21.15.84:0/3858387762 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fb98c09ca40 con 0x7fb99c108e70 2024-04-28T12:47:49.654 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-04-28T12:47:49.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.661+0000 7fb9a0a00640 1 -- 172.21.15.84:0/3858387762 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb9800a3bb0 msgr2=0x7fb9800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:49.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.661+0000 7fb9a0a00640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb9800a3bb0 0x7fb9800a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb988009230 tx=0x7fb9880314a0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.661+0000 7fb9a0a00640 1 -- 172.21.15.84:0/3858387762 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb99c108e70 msgr2=0x7fb99c19f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:49.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.661+0000 7fb9a0a00640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb99c108e70 0x7fb99c19f070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00e9f0 tx=0x7fb98c00eeb0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.661+0000 7fb9a0a00640 1 -- 172.21.15.84:0/3858387762 shutdown_connections 2024-04-28T12:47:49.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.661+0000 7fb9a0a00640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb9800a3bb0 0x7fb9800a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.661+0000 7fb9a0a00640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb99c10a980 0x7fb99c1a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.661+0000 7fb9a0a00640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb99c109fc0 0x7fb99c19f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.661+0000 7fb9a0a00640 1 --2- 172.21.15.84:0/3858387762 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb99c108e70 0x7fb99c19f070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.661+0000 7fb9a0a00640 1 -- 172.21.15.84:0/3858387762 >> 172.21.15.84:0/3858387762 conn(0x7fb99c077e00 msgr2=0x7fb99c10d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:49.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.661+0000 7fb9a0a00640 1 -- 172.21.15.84:0/3858387762 shutdown_connections 2024-04-28T12:47:49.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.661+0000 7fb9a0a00640 1 -- 172.21.15.84:0/3858387762 wait complete. 2024-04-28T12:47:49.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:49.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:49.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:49.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:47:49.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:47:49.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:49.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:49 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1210918782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:47:49.679 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:47:49.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.886+0000 7eff6de00640 1 -- 172.21.15.84:0/189596217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff6810ab40 msgr2=0x7eff681113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:49.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.886+0000 7eff6de00640 1 --2- 172.21.15.84:0/189596217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff6810ab40 0x7eff681113d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7eff60009ca0 tx=0x7eff6002a8c0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.886+0000 7eff6de00640 1 -- 172.21.15.84:0/189596217 shutdown_connections 2024-04-28T12:47:49.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.886+0000 7eff6de00640 1 --2- 172.21.15.84:0/189596217 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff6810ab40 0x7eff681113d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.886+0000 7eff6de00640 1 --2- 172.21.15.84:0/189596217 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff6810a180 0x7eff6810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.886+0000 7eff6de00640 1 --2- 172.21.15.84:0/189596217 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff68108f90 0x7eff681093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.886+0000 7eff6de00640 1 -- 172.21.15.84:0/189596217 >> 172.21.15.84:0/189596217 conn(0x7eff68077f40 msgr2=0x7eff6807a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:49.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.887+0000 7eff6de00640 1 -- 172.21.15.84:0/189596217 shutdown_connections 2024-04-28T12:47:49.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.887+0000 7eff6de00640 1 -- 172.21.15.84:0/189596217 wait complete. 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.888+0000 7eff6de00640 1 Processor -- start 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.888+0000 7eff6de00640 1 -- start start 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.888+0000 7eff6de00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff68108f90 0x7eff6819f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.888+0000 7eff6de00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff6810a180 0x7eff6819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.888+0000 7eff6de00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff6810ab40 0x7eff681a6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.888+0000 7eff6de00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7eff68114b20 con 0x7eff68108f90 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.888+0000 7eff6de00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7eff681149a0 con 0x7eff6810ab40 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.888+0000 7eff6de00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7eff68114ca0 con 0x7eff6810a180 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.889+0000 7eff66a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff68108f90 0x7eff6819f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.889+0000 7eff5fe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff6810a180 0x7eff6819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.889+0000 7eff67400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff6810ab40 0x7eff681a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.889+0000 7eff66a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff68108f90 0x7eff6819f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59110/0 (socket says 172.21.15.84:59110) 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.889+0000 7eff66a00640 1 -- 172.21.15.84:0/262065585 learned_addr learned my addr 172.21.15.84:0/262065585 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.889+0000 7eff5fe00640 1 -- 172.21.15.84:0/262065585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff6810ab40 msgr2=0x7eff681a6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:49.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.889+0000 7eff5fe00640 1 --2- 172.21.15.84:0/262065585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff6810ab40 0x7eff681a6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.889+0000 7eff5fe00640 1 -- 172.21.15.84:0/262065585 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff68108f90 msgr2=0x7eff6819f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:49.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.889+0000 7eff5fe00640 1 --2- 172.21.15.84:0/262065585 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff68108f90 0x7eff6819f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:49.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.889+0000 7eff5fe00640 1 -- 172.21.15.84:0/262065585 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff681a6d80 con 0x7eff6810a180 2024-04-28T12:47:49.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.889+0000 7eff5fe00640 1 --2- 172.21.15.84:0/262065585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff6810a180 0x7eff6819f6a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7eff5000e9f0 tx=0x7eff5000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:49.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.890+0000 7eff6ca00640 1 -- 172.21.15.84:0/262065585 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5000ccc0 con 0x7eff6810a180 2024-04-28T12:47:49.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.890+0000 7eff6ca00640 1 -- 172.21.15.84:0/262065585 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff500108e0 con 0x7eff6810a180 2024-04-28T12:47:49.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.890+0000 7eff6ca00640 1 -- 172.21.15.84:0/262065585 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff50018880 con 0x7eff6810a180 2024-04-28T12:47:49.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.890+0000 7eff6de00640 1 -- 172.21.15.84:0/262065585 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff681a7070 con 0x7eff6810a180 2024-04-28T12:47:49.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.890+0000 7eff6de00640 1 -- 172.21.15.84:0/262065585 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff681a75c0 con 0x7eff6810a180 2024-04-28T12:47:49.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.891+0000 7eff6de00640 1 -- 172.21.15.84:0/262065585 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff68110250 con 0x7eff6810a180 2024-04-28T12:47:49.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.893+0000 7eff6ca00640 1 -- 172.21.15.84:0/262065585 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7eff50010430 con 0x7eff6810a180 2024-04-28T12:47:49.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.894+0000 7eff6ca00640 1 --2- 172.21.15.84:0/262065585 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7eff4c0a3bb0 0x7eff4c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:49.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.894+0000 7eff6ca00640 1 -- 172.21.15.84:0/262065585 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7eff500d02f0 con 0x7eff6810a180 2024-04-28T12:47:49.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.894+0000 7eff66a00640 1 --2- 172.21.15.84:0/262065585 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7eff4c0a3bb0 0x7eff4c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:49.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.895+0000 7eff66a00640 1 --2- 172.21.15.84:0/262065585 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7eff4c0a3bb0 0x7eff4c0a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7eff580058c0 tx=0x7eff58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:49.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:49.898+0000 7eff6ca00640 1 -- 172.21.15.84:0/262065585 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7eff5009ca80 con 0x7eff6810a180 2024-04-28T12:47:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:47:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:47:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:49 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1210918782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:47:49.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:49.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:47:49.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:49.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:47:49.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:47:49.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:49.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:49 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1210918782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:47:50.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.146+0000 7eff6de00640 1 -- 172.21.15.84:0/262065585 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff68063530 con 0x7eff6810a180 2024-04-28T12:47:50.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.147+0000 7eff6ca00640 1 -- 172.21.15.84:0/262065585 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7eff500a0d80 con 0x7eff6810a180 2024-04-28T12:47:50.144 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:47:50.144 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-28T12:47:50.144 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 1 2024-04-28T12:47:50.146 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.150+0000 7eff6de00640 1 -- 172.21.15.84:0/262065585 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7eff4c0a3bb0 msgr2=0x7eff4c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:50.146 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.150+0000 7eff6de00640 1 --2- 172.21.15.84:0/262065585 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7eff4c0a3bb0 0x7eff4c0a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7eff580058c0 tx=0x7eff58005f90 comp rx=0 tx=0).stop 2024-04-28T12:47:50.146 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.150+0000 7eff6de00640 1 -- 172.21.15.84:0/262065585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff6810a180 msgr2=0x7eff6819f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:50.146 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.150+0000 7eff6de00640 1 --2- 172.21.15.84:0/262065585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff6810a180 0x7eff6819f6a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7eff5000e9f0 tx=0x7eff5000eeb0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.147 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.150+0000 7eff6de00640 1 -- 172.21.15.84:0/262065585 shutdown_connections 2024-04-28T12:47:50.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.150+0000 7eff6de00640 1 --2- 172.21.15.84:0/262065585 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7eff4c0a3bb0 0x7eff4c0a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.150+0000 7eff6de00640 1 --2- 172.21.15.84:0/262065585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff6810ab40 0x7eff681a6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.150+0000 7eff6de00640 1 --2- 172.21.15.84:0/262065585 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7eff6810a180 0x7eff6819f6a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.150+0000 7eff6de00640 1 --2- 172.21.15.84:0/262065585 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff68108f90 0x7eff6819f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:50.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.150+0000 7eff6de00640 1 -- 172.21.15.84:0/262065585 >> 172.21.15.84:0/262065585 conn(0x7eff68077f40 msgr2=0x7eff6810d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:50.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.150+0000 7eff6de00640 1 -- 172.21.15.84:0/262065585 shutdown_connections 2024-04-28T12:47:50.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:50.150+0000 7eff6de00640 1 -- 172.21.15.84:0/262065585 wait complete. 2024-04-28T12:47:50.797 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:50 smithi179 ceph-mon[63029]: Deploying daemon mds.l on smithi179 2024-04-28T12:47:50.797 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:50 smithi179 ceph-mon[63029]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:50.797 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:50 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3858387762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:47:50.797 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:50 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/262065585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:47:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:50 smithi084 ceph-mon[28081]: Deploying daemon mds.l on smithi179 2024-04-28T12:47:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:50 smithi084 ceph-mon[28081]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:50 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3858387762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:47:50.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:50 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/262065585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:47:50.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:50 smithi161 ceph-mon[62387]: Deploying daemon mds.l on smithi179 2024-04-28T12:47:50.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:50 smithi161 ceph-mon[62387]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:50.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:50 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3858387762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:47:50.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:50 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/262065585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:47:51.165 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 12:47:50 smithi179 systemd[1]: Starting Ceph mds.l for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:47:51.498 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 12:47:51 smithi179 podman[93087]: 2024-04-28 12:47:51.109577516 +0000 UTC m=+0.018107490 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:47:51.498 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 12:47:51 smithi179 podman[93087]: 2024-04-28 12:47:51.216705926 +0000 UTC m=+0.125235902 container create 2c8bc769d5e5d2f8b320963a2804e3f1e5df21fb442aa607a00ae22129e37700 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-28T12:47:51.498 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 12:47:51 smithi179 podman[93087]: 2024-04-28 12:47:51.503231841 +0000 UTC m=+0.411761814 container init 2c8bc769d5e5d2f8b320963a2804e3f1e5df21fb442aa607a00ae22129e37700 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-28T12:47:51.915 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 12:47:51 smithi179 podman[93087]: 2024-04-28 12:47:51.505859907 +0000 UTC m=+0.414389887 container start 2c8bc769d5e5d2f8b320963a2804e3f1e5df21fb442aa607a00ae22129e37700 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T12:47:51.915 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 12:47:51 smithi179 ceph-mds[93106]: -- 172.21.15.179:0/592582823 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562bc7b65a40 con 0x562bc7b38800 2024-04-28T12:47:51.915 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 12:47:51 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-l[93102]: starting mds.l at 2024-04-28T12:47:51.915 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 12:47:51 smithi179 bash[93087]: 2c8bc769d5e5d2f8b320963a2804e3f1e5df21fb442aa607a00ae22129e37700 2024-04-28T12:47:51.915 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 12:47:51 smithi179 systemd[1]: Started Ceph mds.l for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:47:52.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:47:52.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.384+0000 7fed85400640 1 -- 172.21.15.84:0/4178885947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed80108d90 msgr2=0x7fed801091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:52.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.384+0000 7fed85400640 1 --2- 172.21.15.84:0/4178885947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed80108d90 0x7fed801091b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fed70009900 tx=0x7fed7002a3f0 comp rx=0 tx=0).stop 2024-04-28T12:47:52.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.384+0000 7fed85400640 1 -- 172.21.15.84:0/4178885947 shutdown_connections 2024-04-28T12:47:52.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.384+0000 7fed85400640 1 --2- 172.21.15.84:0/4178885947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed8010a940 0x7fed801111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:52.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.384+0000 7fed85400640 1 --2- 172.21.15.84:0/4178885947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed80109f80 0x7fed8010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:52.382 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.384+0000 7fed85400640 1 --2- 172.21.15.84:0/4178885947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed80108d90 0x7fed801091b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:52.382 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.384+0000 7fed85400640 1 -- 172.21.15.84:0/4178885947 >> 172.21.15.84:0/4178885947 conn(0x7fed80077d30 msgr2=0x7fed8007a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:52.382 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.384+0000 7fed85400640 1 -- 172.21.15.84:0/4178885947 shutdown_connections 2024-04-28T12:47:52.382 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.384+0000 7fed85400640 1 -- 172.21.15.84:0/4178885947 wait complete. 2024-04-28T12:47:52.383 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.385+0000 7fed85400640 1 Processor -- start 2024-04-28T12:47:52.383 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.386+0000 7fed85400640 1 -- start start 2024-04-28T12:47:52.383 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.386+0000 7fed85400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed80108d90 0x7fed8019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:52.383 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.386+0000 7fed85400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed80109f80 0x7fed8019f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.386+0000 7fed85400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed8010a940 0x7fed801a6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.386+0000 7fed85400640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fed801148b0 con 0x7fed80108d90 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.386+0000 7fed85400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fed80114730 con 0x7fed8010a940 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.386+0000 7fed85400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fed80114a30 con 0x7fed80109f80 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.386+0000 7fed7fe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed80108d90 0x7fed8019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.386+0000 7fed7fe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed80108d90 0x7fed8019efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59130/0 (socket says 172.21.15.84:59130) 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.386+0000 7fed7fe00640 1 -- 172.21.15.84:0/2383694594 learned_addr learned my addr 172.21.15.84:0/2383694594 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.386+0000 7fed77400640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed80109f80 0x7fed8019f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.387+0000 7fed84a00640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed8010a940 0x7fed801a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.387+0000 7fed77400640 1 -- 172.21.15.84:0/2383694594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed8010a940 msgr2=0x7fed801a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.387+0000 7fed77400640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed8010a940 0x7fed801a6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.387+0000 7fed77400640 1 -- 172.21.15.84:0/2383694594 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed80108d90 msgr2=0x7fed8019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.387+0000 7fed77400640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed80108d90 0x7fed8019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:52.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.387+0000 7fed77400640 1 -- 172.21.15.84:0/2383694594 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed801a6bf0 con 0x7fed80109f80 2024-04-28T12:47:52.385 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.387+0000 7fed77400640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed80109f80 0x7fed8019f510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fed6800e9f0 tx=0x7fed6800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:52.385 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.387+0000 7fed7d600640 1 -- 172.21.15.84:0/2383694594 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6800ccc0 con 0x7fed80109f80 2024-04-28T12:47:52.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.387+0000 7fed7d600640 1 -- 172.21.15.84:0/2383694594 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed68005020 con 0x7fed80109f80 2024-04-28T12:47:52.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.387+0000 7fed7d600640 1 -- 172.21.15.84:0/2383694594 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed68004a00 con 0x7fed80109f80 2024-04-28T12:47:52.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.387+0000 7fed85400640 1 -- 172.21.15.84:0/2383694594 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed801a6ee0 con 0x7fed80109f80 2024-04-28T12:47:52.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.388+0000 7fed85400640 1 -- 172.21.15.84:0/2383694594 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed801a73a0 con 0x7fed80109f80 2024-04-28T12:47:52.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.389+0000 7fed85400640 1 -- 172.21.15.84:0/2383694594 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed801101d0 con 0x7fed80109f80 2024-04-28T12:47:52.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.390+0000 7fed7d600640 1 -- 172.21.15.84:0/2383694594 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fed68004ba0 con 0x7fed80109f80 2024-04-28T12:47:52.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.392+0000 7fed7d600640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed640a3c30 0x7fed640a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:52.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.392+0000 7fed7d600640 1 -- 172.21.15.84:0/2383694594 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fed680d00c0 con 0x7fed80109f80 2024-04-28T12:47:52.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.392+0000 7fed7fe00640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed640a3c30 0x7fed640a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:52.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.393+0000 7fed7fe00640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed640a3c30 0x7fed640a60f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fed70000c00 tx=0x7fed700314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:52.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.394+0000 7fed7d600640 1 -- 172.21.15.84:0/2383694594 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fed6809cea0 con 0x7fed80109f80 2024-04-28T12:47:52.609 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:52 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.179:6832/2409757407,v1:172.21.15.179:6833/2409757407] up:boot 2024-04-28T12:47:52.609 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:52 smithi161 ceph-mon[62387]: fsmap 1 up:standby 2024-04-28T12:47:52.609 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-28T12:47:52.609 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:52 smithi161 ceph-mon[62387]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:52.609 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:52.609 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:52.609 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:52.609 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:47:52.609 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:47:52.609 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:52.635 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.639+0000 7fed85400640 1 -- 172.21.15.84:0/2383694594 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed8002dad0 con 0x7fed80109f80 2024-04-28T12:47:52.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.640+0000 7fed7d600640 1 -- 172.21.15.84:0/2383694594 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fed6809c7a0 con 0x7fed80109f80 2024-04-28T12:47:52.636 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:47:52.637 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-28T12:47:52.637 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 2 2024-04-28T12:47:52.639 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.643+0000 7fed85400640 1 -- 172.21.15.84:0/2383694594 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed640a3c30 msgr2=0x7fed640a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:52.639 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.643+0000 7fed85400640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed640a3c30 0x7fed640a60f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fed70000c00 tx=0x7fed700314a0 comp rx=0 tx=0).stop 2024-04-28T12:47:52.640 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.643+0000 7fed85400640 1 -- 172.21.15.84:0/2383694594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed80109f80 msgr2=0x7fed8019f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:52.640 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.643+0000 7fed85400640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed80109f80 0x7fed8019f510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fed6800e9f0 tx=0x7fed6800eeb0 comp rx=0 tx=0).stop 2024-04-28T12:47:52.640 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.643+0000 7fed85400640 1 -- 172.21.15.84:0/2383694594 shutdown_connections 2024-04-28T12:47:52.640 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.643+0000 7fed85400640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed640a3c30 0x7fed640a60f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:52.640 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.643+0000 7fed85400640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed8010a940 0x7fed801a6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:52.640 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.643+0000 7fed85400640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed80109f80 0x7fed8019f510 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:52.640 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.643+0000 7fed85400640 1 --2- 172.21.15.84:0/2383694594 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed80108d90 0x7fed8019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:52.640 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.643+0000 7fed85400640 1 -- 172.21.15.84:0/2383694594 >> 172.21.15.84:0/2383694594 conn(0x7fed80077d30 msgr2=0x7fed8010d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:52.640 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.643+0000 7fed85400640 1 -- 172.21.15.84:0/2383694594 shutdown_connections 2024-04-28T12:47:52.640 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:52.643+0000 7fed85400640 1 -- 172.21.15.84:0/2383694594 wait complete. 2024-04-28T12:47:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:52 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.179:6832/2409757407,v1:172.21.15.179:6833/2409757407] up:boot 2024-04-28T12:47:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:52 smithi179 ceph-mon[63029]: fsmap 1 up:standby 2024-04-28T12:47:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-28T12:47:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:52 smithi179 ceph-mon[63029]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:47:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:47:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:52.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:52 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.179:6832/2409757407,v1:172.21.15.179:6833/2409757407] up:boot 2024-04-28T12:47:52.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:52 smithi084 ceph-mon[28081]: fsmap 1 up:standby 2024-04-28T12:47:52.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-28T12:47:52.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:52 smithi084 ceph-mon[28081]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:52.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:52.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:52.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:52.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:47:52.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:47:52.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:53.848 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:53 smithi161 ceph-mon[62387]: Deploying daemon mds.k on smithi161 2024-04-28T12:47:53.848 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:53 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2383694594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:47:53.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:53 smithi179 ceph-mon[63029]: Deploying daemon mds.k on smithi161 2024-04-28T12:47:53.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:53 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2383694594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:47:53.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:53 smithi084 ceph-mon[28081]: Deploying daemon mds.k on smithi161 2024-04-28T12:47:53.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:53 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2383694594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:47:54.232 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 12:47:54 smithi161 systemd[1]: Starting Ceph mds.k for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:47:54.492 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 12:47:54 smithi161 podman[91510]: 2024-04-28 12:47:54.138853539 +0000 UTC m=+0.017721604 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:47:54.492 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 12:47:54 smithi161 podman[91510]: 2024-04-28 12:47:54.267879647 +0000 UTC m=+0.146747688 container create 5dbf90dedeaeca2431c0a93bb0e56b4b5a089c03db545e30ec2e662309e802d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD) 2024-04-28T12:47:54.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:47:54.866 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.869+0000 7f207e200640 1 -- 172.21.15.84:0/1780331469 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20780772f0 msgr2=0x7f2078077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:54.866 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.869+0000 7f207e200640 1 --2- 172.21.15.84:0/1780331469 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20780772f0 0x7f2078077710 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2068009900 tx=0x7f206802a410 comp rx=0 tx=0).stop 2024-04-28T12:47:54.866 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.869+0000 7f207e200640 1 -- 172.21.15.84:0/1780331469 shutdown_connections 2024-04-28T12:47:54.866 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.870+0000 7f207e200640 1 --2- 172.21.15.84:0/1780331469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2078078ed0 0x7f2078079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.867 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.870+0000 7f207e200640 1 --2- 172.21.15.84:0/1780331469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2078078530 0x7f2078078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.867 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.870+0000 7f207e200640 1 --2- 172.21.15.84:0/1780331469 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20780772f0 0x7f2078077710 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.867 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.870+0000 7f207e200640 1 -- 172.21.15.84:0/1780331469 >> 172.21.15.84:0/1780331469 conn(0x7f2078100120 msgr2=0x7f2078102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:54.867 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.870+0000 7f207e200640 1 -- 172.21.15.84:0/1780331469 shutdown_connections 2024-04-28T12:47:54.867 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.870+0000 7f207e200640 1 -- 172.21.15.84:0/1780331469 wait complete. 2024-04-28T12:47:54.868 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.871+0000 7f207e200640 1 Processor -- start 2024-04-28T12:47:54.868 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.871+0000 7f207e200640 1 -- start start 2024-04-28T12:47:54.868 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.871+0000 7f207e200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20780772f0 0x7f207819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.871+0000 7f207e200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2078078530 0x7f207819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.871+0000 7f207e200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2078078ed0 0x7f20781a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.871+0000 7f207e200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2078114af0 con 0x7f20780772f0 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.871+0000 7f207e200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2078114970 con 0x7f2078078ed0 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.871+0000 7f207e200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2078114c70 con 0x7f2078078530 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.871+0000 7f2077400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20780772f0 0x7f207819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.871+0000 7f2077400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20780772f0 0x7f207819f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59146/0 (socket says 172.21.15.84:59146) 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.871+0000 7f2077400640 1 -- 172.21.15.84:0/3584638332 learned_addr learned my addr 172.21.15.84:0/3584638332 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.872+0000 7f2076a00640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2078078530 0x7f207819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.872+0000 7f2077e00640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2078078ed0 0x7f20781a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.872+0000 7f2077400640 1 -- 172.21.15.84:0/3584638332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2078078530 msgr2=0x7f207819f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.872+0000 7f2077400640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2078078530 0x7f207819f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.872+0000 7f2077400640 1 -- 172.21.15.84:0/3584638332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2078078ed0 msgr2=0x7f20781a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.872+0000 7f2077400640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2078078ed0 0x7f20781a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:54.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.872+0000 7f2077400640 1 -- 172.21.15.84:0/3584638332 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20781a6dc0 con 0x7f20780772f0 2024-04-28T12:47:54.870 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.872+0000 7f2077400640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20780772f0 0x7f207819f1a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2068005ab0 tx=0x7f2068003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:54.870 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.872+0000 7f206fe00640 1 -- 172.21.15.84:0/3584638332 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206803f070 con 0x7f20780772f0 2024-04-28T12:47:54.870 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.873+0000 7f206fe00640 1 -- 172.21.15.84:0/3584638332 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2068031c50 con 0x7f20780772f0 2024-04-28T12:47:54.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.873+0000 7f206fe00640 1 -- 172.21.15.84:0/3584638332 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2068035030 con 0x7f20780772f0 2024-04-28T12:47:54.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.873+0000 7f207e200640 1 -- 172.21.15.84:0/3584638332 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20781a7050 con 0x7f20780772f0 2024-04-28T12:47:54.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.873+0000 7f207e200640 1 -- 172.21.15.84:0/3584638332 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20781a74f0 con 0x7f20780772f0 2024-04-28T12:47:54.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.874+0000 7f207e200640 1 -- 172.21.15.84:0/3584638332 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20781114e0 con 0x7f20780772f0 2024-04-28T12:47:54.872 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.876+0000 7f206fe00640 1 -- 172.21.15.84:0/3584638332 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2068042070 con 0x7f20780772f0 2024-04-28T12:47:54.873 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.877+0000 7f206fe00640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f205c0a3c30 0x7f205c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:54.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.878+0000 7f206fe00640 1 -- 172.21.15.84:0/3584638332 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f20680f05c0 con 0x7f20780772f0 2024-04-28T12:47:54.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.878+0000 7f2076a00640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f205c0a3c30 0x7f205c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:54.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.878+0000 7f2076a00640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f205c0a3c30 0x7f205c0a60f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f20781a06b0 tx=0x7f2064005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:54.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:54.882+0000 7f206fe00640 1 -- 172.21.15.84:0/3584638332 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f20680bcdf0 con 0x7f20780772f0 2024-04-28T12:47:54.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:54 smithi179 ceph-mon[63029]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:54.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:54 smithi084 ceph-mon[28081]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:54.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:54 smithi161 ceph-mon[62387]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:54.992 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 12:47:54 smithi161 podman[91510]: 2024-04-28 12:47:54.537360152 +0000 UTC m=+0.416228193 container init 5dbf90dedeaeca2431c0a93bb0e56b4b5a089c03db545e30ec2e662309e802d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T12:47:54.992 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 12:47:54 smithi161 podman[91510]: 2024-04-28 12:47:54.539872743 +0000 UTC m=+0.418740786 container start 5dbf90dedeaeca2431c0a93bb0e56b4b5a089c03db545e30ec2e662309e802d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-28T12:47:54.992 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 12:47:54 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-k[91525]: starting mds.k at 2024-04-28T12:47:54.993 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 12:47:54 smithi161 bash[91510]: 5dbf90dedeaeca2431c0a93bb0e56b4b5a089c03db545e30ec2e662309e802d3 2024-04-28T12:47:54.993 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 12:47:54 smithi161 systemd[1]: Started Ceph mds.k for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:47:55.125 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.129+0000 7f207e200640 1 -- 172.21.15.84:0/3584638332 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f207802dad0 con 0x7f20780772f0 2024-04-28T12:47:55.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.130+0000 7f206fe00640 1 -- 172.21.15.84:0/3584638332 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f20680c10f0 con 0x7f20780772f0 2024-04-28T12:47:55.127 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:47:55.127 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-28T12:47:55.127 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 2 2024-04-28T12:47:55.129 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.133+0000 7f207e200640 1 -- 172.21.15.84:0/3584638332 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f205c0a3c30 msgr2=0x7f205c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:55.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.133+0000 7f207e200640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f205c0a3c30 0x7f205c0a60f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f20781a06b0 tx=0x7f2064005f90 comp rx=0 tx=0).stop 2024-04-28T12:47:55.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.133+0000 7f207e200640 1 -- 172.21.15.84:0/3584638332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20780772f0 msgr2=0x7f207819f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:55.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.133+0000 7f207e200640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20780772f0 0x7f207819f1a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2068005ab0 tx=0x7f2068003b90 comp rx=0 tx=0).stop 2024-04-28T12:47:55.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.133+0000 7f207e200640 1 -- 172.21.15.84:0/3584638332 shutdown_connections 2024-04-28T12:47:55.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.133+0000 7f207e200640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f205c0a3c30 0x7f205c0a60f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:55.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.133+0000 7f207e200640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2078078ed0 0x7f20781a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:55.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.133+0000 7f207e200640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2078078530 0x7f207819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:55.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.133+0000 7f207e200640 1 --2- 172.21.15.84:0/3584638332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f20780772f0 0x7f207819f1a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:55.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.133+0000 7f207e200640 1 -- 172.21.15.84:0/3584638332 >> 172.21.15.84:0/3584638332 conn(0x7f2078100120 msgr2=0x7f2078101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:55.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.133+0000 7f207e200640 1 -- 172.21.15.84:0/3584638332 shutdown_connections 2024-04-28T12:47:55.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:55.133+0000 7f207e200640 1 -- 172.21.15.84:0/3584638332 wait complete. 2024-04-28T12:47:55.940 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:55.940 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:55.940 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:55.940 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:47:55.940 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:47:55.940 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:55.940 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:55 smithi084 ceph-mon[28081]: Deploying daemon mds.j on smithi084 2024-04-28T12:47:55.940 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:55 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3584638332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:47:55.940 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:55 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.161:6832/3034002883,v1:172.21.15.161:6833/3034002883] up:boot 2024-04-28T12:47:55.940 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:55 smithi084 ceph-mon[28081]: fsmap 2 up:standby 2024-04-28T12:47:55.941 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-28T12:47:55.941 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:55 smithi084 ceph-mon[28081]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:55.952 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:55.952 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:55.952 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:55.952 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:47:55.952 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:47:55.952 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:55.952 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:55 smithi161 ceph-mon[62387]: Deploying daemon mds.j on smithi084 2024-04-28T12:47:55.953 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:55 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3584638332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:47:55.953 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:55 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.161:6832/3034002883,v1:172.21.15.161:6833/3034002883] up:boot 2024-04-28T12:47:55.953 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:55 smithi161 ceph-mon[62387]: fsmap 2 up:standby 2024-04-28T12:47:55.953 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-28T12:47:55.953 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:55 smithi161 ceph-mon[62387]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:47:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:47:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:55 smithi179 ceph-mon[63029]: Deploying daemon mds.j on smithi084 2024-04-28T12:47:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:55 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3584638332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:47:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:55 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.161:6832/3034002883,v1:172.21.15.161:6833/3034002883] up:boot 2024-04-28T12:47:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:55 smithi179 ceph-mon[63029]: fsmap 2 up:standby 2024-04-28T12:47:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-28T12:47:56.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:55 smithi179 ceph-mon[63029]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:57.149 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:47:57.539 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.542+0000 7f7724a00640 1 -- 172.21.15.84:0/854012783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7718095d90 msgr2=0x7f77180961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:57.539 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.542+0000 7f7724a00640 1 --2- 172.21.15.84:0/854012783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7718095d90 0x7f77180961b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7714009900 tx=0x7f771402a3f0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.539 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.542+0000 7f7724a00640 1 -- 172.21.15.84:0/854012783 shutdown_connections 2024-04-28T12:47:57.539 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.542+0000 7f7724a00640 1 --2- 172.21.15.84:0/854012783 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7718097940 0x7f771809e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.539 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.542+0000 7f7724a00640 1 --2- 172.21.15.84:0/854012783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7718096f80 0x7f7718097400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.539 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.542+0000 7f7724a00640 1 --2- 172.21.15.84:0/854012783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7718095d90 0x7f77180961b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.539 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.542+0000 7f7724a00640 1 -- 172.21.15.84:0/854012783 >> 172.21.15.84:0/854012783 conn(0x7f7718091300 msgr2=0x7f7718093740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:57.539 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.542+0000 7f7724a00640 1 -- 172.21.15.84:0/854012783 shutdown_connections 2024-04-28T12:47:57.539 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.542+0000 7f7724a00640 1 -- 172.21.15.84:0/854012783 wait complete. 2024-04-28T12:47:57.541 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.543+0000 7f7724a00640 1 Processor -- start 2024-04-28T12:47:57.541 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.543+0000 7f7724a00640 1 -- start start 2024-04-28T12:47:57.541 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.543+0000 7f7724a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7718095d90 0x7f771812e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.543+0000 7f7724a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7718096f80 0x7f771812e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.543+0000 7f7724a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7718097940 0x7f7718135750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.543+0000 7f7724a00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f77180a18f0 con 0x7f7718097940 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.543+0000 7f7724a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f77180a1770 con 0x7f7718096f80 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.543+0000 7f7724a00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f77180a1a70 con 0x7f7718095d90 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.544+0000 7f771ea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7718096f80 0x7f771812e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.544+0000 7f771ea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7718096f80 0x7f771812e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:55230/0 (socket says 172.21.15.84:55230) 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.544+0000 7f771ea00640 1 -- 172.21.15.84:0/2795565603 learned_addr learned my addr 172.21.15.84:0/2795565603 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.544+0000 7f771f400640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7718095d90 0x7f771812e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.544+0000 7f771fe00640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7718097940 0x7f7718135750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.544+0000 7f771ea00640 1 -- 172.21.15.84:0/2795565603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7718095d90 msgr2=0x7f771812e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.544+0000 7f771ea00640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7718095d90 0x7f771812e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.544+0000 7f771ea00640 1 -- 172.21.15.84:0/2795565603 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7718097940 msgr2=0x7f7718135750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:57.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.544+0000 7f771ea00640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7718097940 0x7f7718135750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.544+0000 7f771ea00640 1 -- 172.21.15.84:0/2795565603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7718135dc0 con 0x7f7718096f80 2024-04-28T12:47:57.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.545+0000 7f771ea00640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7718096f80 0x7f771812e6e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f770c00e9f0 tx=0x7f770c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:57.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.545+0000 7f7713e00640 1 -- 172.21.15.84:0/2795565603 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c00cd80 con 0x7f7718096f80 2024-04-28T12:47:57.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.545+0000 7f7713e00640 1 -- 172.21.15.84:0/2795565603 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f770c0108e0 con 0x7f7718096f80 2024-04-28T12:47:57.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.545+0000 7f7724a00640 1 -- 172.21.15.84:0/2795565603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77181360b0 con 0x7f7718096f80 2024-04-28T12:47:57.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.545+0000 7f7713e00640 1 -- 172.21.15.84:0/2795565603 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c018980 con 0x7f7718096f80 2024-04-28T12:47:57.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.545+0000 7f7724a00640 1 -- 172.21.15.84:0/2795565603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7718136600 con 0x7f7718096f80 2024-04-28T12:47:57.544 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.546+0000 7f7724a00640 1 -- 172.21.15.84:0/2795565603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7718136860 con 0x7f7718096f80 2024-04-28T12:47:57.544 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.548+0000 7f7713e00640 1 -- 172.21.15.84:0/2795565603 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f770c010430 con 0x7f7718096f80 2024-04-28T12:47:57.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.549+0000 7f7713e00640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f77080a3bb0 0x7f77080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:47:57.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.550+0000 7f7713e00640 1 -- 172.21.15.84:0/2795565603 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f770c0d0380 con 0x7f7718096f80 2024-04-28T12:47:57.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.550+0000 7f771f400640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f77080a3bb0 0x7f77080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:47:57.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.551+0000 7f771f400640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f77080a3bb0 0x7f77080a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7714009230 tx=0x7f77140314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:47:57.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.554+0000 7f7713e00640 1 -- 172.21.15.84:0/2795565603 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f770c09d160 con 0x7f7718096f80 2024-04-28T12:47:57.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.815+0000 7f7724a00640 1 -- 172.21.15.84:0/2795565603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7718136a30 con 0x7f7718096f80 2024-04-28T12:47:57.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.816+0000 7f7713e00640 1 -- 172.21.15.84:0/2795565603 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f770c09ca60 con 0x7f7718096f80 2024-04-28T12:47:57.812 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:47:57.812 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-28T12:47:57.813 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 3 2024-04-28T12:47:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.819+0000 7f7724a00640 1 -- 172.21.15.84:0/2795565603 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f77080a3bb0 msgr2=0x7f77080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.819+0000 7f7724a00640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f77080a3bb0 0x7f77080a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7714009230 tx=0x7f77140314a0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.819+0000 7f7724a00640 1 -- 172.21.15.84:0/2795565603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7718096f80 msgr2=0x7f771812e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:47:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.819+0000 7f7724a00640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7718096f80 0x7f771812e6e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f770c00e9f0 tx=0x7f770c00eeb0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.819+0000 7f7724a00640 1 -- 172.21.15.84:0/2795565603 shutdown_connections 2024-04-28T12:47:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.819+0000 7f7724a00640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f77080a3bb0 0x7f77080a6070 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.819+0000 7f7724a00640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7718097940 0x7f7718135750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.819+0000 7f7724a00640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7718096f80 0x7f771812e6e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.819+0000 7f7724a00640 1 --2- 172.21.15.84:0/2795565603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7718095d90 0x7f771812e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:47:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.819+0000 7f7724a00640 1 -- 172.21.15.84:0/2795565603 >> 172.21.15.84:0/2795565603 conn(0x7f7718091300 msgr2=0x7f7718006d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:47:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.819+0000 7f7724a00640 1 -- 172.21.15.84:0/2795565603 shutdown_connections 2024-04-28T12:47:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:47:57.819+0000 7f7724a00640 1 -- 172.21.15.84:0/2795565603 wait complete. 2024-04-28T12:47:58.417 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 12:47:58 smithi084 podman[65125]: 2024-04-28 12:47:58.166270959 +0000 UTC m=+0.177219371 container create 80c5ddf586152c5553058c80caba116619d8d81f6b08d315617f30fbb461caea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-28T12:47:58.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:58 smithi179 ceph-mon[63029]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:58.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:58 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2795565603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:47:58.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:58 smithi084 ceph-mon[28081]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:58.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:58 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2795565603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:47:58.917 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 12:47:58 smithi084 podman[65125]: 2024-04-28 12:47:58.486235338 +0000 UTC m=+0.497183833 container init 80c5ddf586152c5553058c80caba116619d8d81f6b08d315617f30fbb461caea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-28T12:47:58.917 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 12:47:58 smithi084 podman[65125]: 2024-04-28 12:47:58.496450882 +0000 UTC m=+0.507399301 container start 80c5ddf586152c5553058c80caba116619d8d81f6b08d315617f30fbb461caea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-28T12:47:58.917 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 12:47:58 smithi084 ceph-mds[65144]: -- 172.21.15.84:0/2678558457 <== mon.0 v2:172.21.15.84:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e36b6f5a40 con 0x55e36b6c8800 2024-04-28T12:47:58.917 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 12:47:58 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-j[65140]: starting mds.j at 2024-04-28T12:47:58.917 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 12:47:58 smithi084 bash[65125]: 80c5ddf586152c5553058c80caba116619d8d81f6b08d315617f30fbb461caea 2024-04-28T12:47:58.917 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 12:47:58 smithi084 systemd[1]: Started Ceph mds.j for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:47:58.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:58 smithi161 ceph-mon[62387]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:47:58.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:58 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2795565603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:47:59.836 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:47:59.914 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:59 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.84:6834/3549575919,v1:172.21.15.84:6835/3549575919] up:boot 2024-04-28T12:47:59.914 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:59 smithi084 ceph-mon[28081]: fsmap 3 up:standby 2024-04-28T12:47:59.914 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-28T12:47:59.914 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:59.914 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:59.914 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:59.914 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:47:59.914 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:47:59.914 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:47:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:59.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:59 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.84:6834/3549575919,v1:172.21.15.84:6835/3549575919] up:boot 2024-04-28T12:47:59.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:59 smithi179 ceph-mon[63029]: fsmap 3 up:standby 2024-04-28T12:47:59.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:59 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-28T12:47:59.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:59 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:59.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:59 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:59.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:59 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:59.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:59 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:47:59.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:59 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:47:59.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:47:59 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:47:59.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:59 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.84:6834/3549575919,v1:172.21.15.84:6835/3549575919] up:boot 2024-04-28T12:47:59.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:59 smithi161 ceph-mon[62387]: fsmap 3 up:standby 2024-04-28T12:47:59.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:59 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-28T12:47:59.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:59 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:59.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:59 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:59.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:59 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:47:59.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:59 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:47:59.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:59 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:47:59.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:47:59 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:00.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.059+0000 7fb948c00640 1 -- 172.21.15.84:0/3892490035 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb944106540 msgr2=0x7fb94410cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:00.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.059+0000 7fb948c00640 1 --2- 172.21.15.84:0/3892490035 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb944106540 0x7fb94410cdd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009ca0 tx=0x7fb93c02a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.059+0000 7fb948c00640 1 -- 172.21.15.84:0/3892490035 shutdown_connections 2024-04-28T12:48:00.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.059+0000 7fb948c00640 1 --2- 172.21.15.84:0/3892490035 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb944106540 0x7fb94410cdd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.059+0000 7fb948c00640 1 --2- 172.21.15.84:0/3892490035 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb944105b80 0x7fb944106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.059+0000 7fb948c00640 1 --2- 172.21.15.84:0/3892490035 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb944104990 0x7fb944104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.059+0000 7fb948c00640 1 -- 172.21.15.84:0/3892490035 >> 172.21.15.84:0/3892490035 conn(0x7fb9440fff00 msgr2=0x7fb944102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:00.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.059+0000 7fb948c00640 1 -- 172.21.15.84:0/3892490035 shutdown_connections 2024-04-28T12:48:00.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.059+0000 7fb948c00640 1 -- 172.21.15.84:0/3892490035 wait complete. 2024-04-28T12:48:00.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.060+0000 7fb948c00640 1 Processor -- start 2024-04-28T12:48:00.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.061+0000 7fb948c00640 1 -- start start 2024-04-28T12:48:00.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.061+0000 7fb948c00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb944104990 0x7fb94419ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:00.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.061+0000 7fb948c00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb944105b80 0x7fb94419b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:00.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.061+0000 7fb948c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb944106540 0x7fb9441a2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:00.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.061+0000 7fb948c00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb944110510 con 0x7fb944104990 2024-04-28T12:48:00.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.061+0000 7fb948c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb944110390 con 0x7fb944106540 2024-04-28T12:48:00.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.061+0000 7fb948c00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb944110690 con 0x7fb944105b80 2024-04-28T12:48:00.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.061+0000 7fb943400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb944104990 0x7fb94419ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:00.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.061+0000 7fb943400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb944104990 0x7fb94419ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36454/0 (socket says 172.21.15.84:36454) 2024-04-28T12:48:00.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.061+0000 7fb943400640 1 -- 172.21.15.84:0/4230257814 learned_addr learned my addr 172.21.15.84:0/4230257814 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:00.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.061+0000 7fb93be00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb944105b80 0x7fb94419b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:00.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb943e00640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb944106540 0x7fb9441a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:00.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb93be00640 1 -- 172.21.15.84:0/4230257814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb944106540 msgr2=0x7fb9441a2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:00.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb93be00640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb944106540 0x7fb9441a2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb93be00640 1 -- 172.21.15.84:0/4230257814 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb944104990 msgr2=0x7fb94419ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:00.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb93be00640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb944104990 0x7fb94419ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb93be00640 1 -- 172.21.15.84:0/4230257814 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9441a2770 con 0x7fb944105b80 2024-04-28T12:48:00.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb943e00640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb944106540 0x7fb9441a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:00.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb943400640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb944104990 0x7fb94419ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:00.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb93be00640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb944105b80 0x7fb94419b090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00e9f0 tx=0x7fb92c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:00.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb940c00640 1 -- 172.21.15.84:0/4230257814 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92c00ccc0 con 0x7fb944105b80 2024-04-28T12:48:00.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb940c00640 1 -- 172.21.15.84:0/4230257814 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb92c005020 con 0x7fb944105b80 2024-04-28T12:48:00.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb948c00640 1 -- 172.21.15.84:0/4230257814 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9441a2a60 con 0x7fb944105b80 2024-04-28T12:48:00.064 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb940c00640 1 -- 172.21.15.84:0/4230257814 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92c004a00 con 0x7fb944105b80 2024-04-28T12:48:00.064 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.062+0000 7fb948c00640 1 -- 172.21.15.84:0/4230257814 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9441a2f60 con 0x7fb944105b80 2024-04-28T12:48:00.064 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.064+0000 7fb948c00640 1 -- 172.21.15.84:0/4230257814 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb94410bcb0 con 0x7fb944105b80 2024-04-28T12:48:00.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.066+0000 7fb940c00640 1 -- 172.21.15.84:0/4230257814 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb92c004ba0 con 0x7fb944105b80 2024-04-28T12:48:00.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.067+0000 7fb940c00640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb9280a3bb0 0x7fb9280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:00.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.067+0000 7fb940c00640 1 -- 172.21.15.84:0/4230257814 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb92c0cfff0 con 0x7fb944105b80 2024-04-28T12:48:00.066 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.067+0000 7fb943400640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb9280a3bb0 0x7fb9280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:00.066 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.068+0000 7fb943400640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb9280a3bb0 0x7fb9280a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb9340058c0 tx=0x7fb934005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:00.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.071+0000 7fb940c00640 1 -- 172.21.15.84:0/4230257814 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb92c09cdf0 con 0x7fb944105b80 2024-04-28T12:48:00.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.322+0000 7fb948c00640 1 -- 172.21.15.84:0/4230257814 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb944063530 con 0x7fb944105b80 2024-04-28T12:48:00.319 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.323+0000 7fb940c00640 1 -- 172.21.15.84:0/4230257814 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fb92c09c6f0 con 0x7fb944105b80 2024-04-28T12:48:00.319 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:00.319 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-28T12:48:00.320 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 4 2024-04-28T12:48:00.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.325+0000 7fb948c00640 1 -- 172.21.15.84:0/4230257814 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb9280a3bb0 msgr2=0x7fb9280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:00.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.325+0000 7fb948c00640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb9280a3bb0 0x7fb9280a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb9340058c0 tx=0x7fb934005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:00.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.325+0000 7fb948c00640 1 -- 172.21.15.84:0/4230257814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb944105b80 msgr2=0x7fb94419b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:00.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.325+0000 7fb948c00640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb944105b80 0x7fb94419b090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00e9f0 tx=0x7fb92c00eeb0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.326+0000 7fb948c00640 1 -- 172.21.15.84:0/4230257814 shutdown_connections 2024-04-28T12:48:00.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.326+0000 7fb948c00640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb9280a3bb0 0x7fb9280a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.326+0000 7fb948c00640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb944106540 0x7fb9441a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.326+0000 7fb948c00640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb944105b80 0x7fb94419b090 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.326+0000 7fb948c00640 1 --2- 172.21.15.84:0/4230257814 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb944104990 0x7fb94419ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:00.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.326+0000 7fb948c00640 1 -- 172.21.15.84:0/4230257814 >> 172.21.15.84:0/4230257814 conn(0x7fb9440fff00 msgr2=0x7fb944108e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:00.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.326+0000 7fb948c00640 1 -- 172.21.15.84:0/4230257814 shutdown_connections 2024-04-28T12:48:00.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:00.326+0000 7fb948c00640 1 -- 172.21.15.84:0/4230257814 wait complete. 2024-04-28T12:48:00.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:00 smithi179 ceph-mon[63029]: Deploying daemon mds.d on smithi084 2024-04-28T12:48:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:00 smithi179 ceph-mon[63029]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:00 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4230257814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:00.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:00 smithi084 ceph-mon[28081]: Deploying daemon mds.d on smithi084 2024-04-28T12:48:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:00 smithi084 ceph-mon[28081]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:00.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:00 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4230257814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:00.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:00 smithi161 ceph-mon[62387]: Deploying daemon mds.d on smithi084 2024-04-28T12:48:00.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:00 smithi161 ceph-mon[62387]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:00.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:00 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4230257814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:02.152 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 12:48:01 smithi084 systemd[1]: Starting Ceph mds.d for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:48:02.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:02.413 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 12:48:02 smithi084 podman[65468]: 2024-04-28 12:48:02.061156957 +0000 UTC m=+0.034840750 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:02.413 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 12:48:02 smithi084 podman[65468]: 2024-04-28 12:48:02.196017563 +0000 UTC m=+0.169701381 container create a1ee823846e83887fd1c06b05a883b6fa4281c2d430af607a52bbfc423145871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T12:48:02.545 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.548+0000 7fb0b3e00640 1 -- 172.21.15.84:0/1163907593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0b410a660 msgr2=0x7fb0b410caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:02.545 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.548+0000 7fb0b3e00640 1 --2- 172.21.15.84:0/1163907593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0b410a660 0x7fb0b410caf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009ca0 tx=0x7fb0ac02a950 comp rx=0 tx=0).stop 2024-04-28T12:48:02.545 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.548+0000 7fb0b3e00640 1 -- 172.21.15.84:0/1163907593 shutdown_connections 2024-04-28T12:48:02.545 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.548+0000 7fb0b3e00640 1 --2- 172.21.15.84:0/1163907593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0b410a660 0x7fb0b410caf0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:02.545 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.548+0000 7fb0b3e00640 1 --2- 172.21.15.84:0/1163907593 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0b4075310 0x7fb0b4075770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:02.545 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.548+0000 7fb0b3e00640 1 --2- 172.21.15.84:0/1163907593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0b40768f0 0x7fb0b4074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:02.545 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.548+0000 7fb0b3e00640 1 -- 172.21.15.84:0/1163907593 >> 172.21.15.84:0/1163907593 conn(0x7fb0b40fdd40 msgr2=0x7fb0b4100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:02.545 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.548+0000 7fb0b3e00640 1 -- 172.21.15.84:0/1163907593 shutdown_connections 2024-04-28T12:48:02.545 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.548+0000 7fb0b3e00640 1 -- 172.21.15.84:0/1163907593 wait complete. 2024-04-28T12:48:02.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.549+0000 7fb0b3e00640 1 Processor -- start 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.549+0000 7fb0b3e00640 1 -- start start 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.549+0000 7fb0b3e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0b4075310 0x7fb0b419ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.549+0000 7fb0b3e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0b40768f0 0x7fb0b419b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.549+0000 7fb0b3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0b410a660 0x7fb0b41a2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0b3e00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b4110540 con 0x7fb0b4075310 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0b3e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41103c0 con 0x7fb0b410a660 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0b3e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41106c0 con 0x7fb0b40768f0 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0b2a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0b4075310 0x7fb0b419ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0b2a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0b4075310 0x7fb0b419ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36502/0 (socket says 172.21.15.84:36502) 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0b2a00640 1 -- 172.21.15.84:0/702991053 learned_addr learned my addr 172.21.15.84:0/702991053 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0b3400640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0b410a660 0x7fb0b41a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0abe00640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0b40768f0 0x7fb0b419b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0b2a00640 1 -- 172.21.15.84:0/702991053 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0b40768f0 msgr2=0x7fb0b419b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:02.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0b2a00640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0b40768f0 0x7fb0b419b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:02.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0b2a00640 1 -- 172.21.15.84:0/702991053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0b410a660 msgr2=0x7fb0b41a2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:02.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0b2a00640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0b410a660 0x7fb0b41a2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:02.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.550+0000 7fb0b2a00640 1 -- 172.21.15.84:0/702991053 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b41a2770 con 0x7fb0b4075310 2024-04-28T12:48:02.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.551+0000 7fb0b2a00640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0b4075310 0x7fb0b419ab50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400e9f0 tx=0x7fb0a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:02.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.551+0000 7fb0ab400640 1 -- 172.21.15.84:0/702991053 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a400cd80 con 0x7fb0b4075310 2024-04-28T12:48:02.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.551+0000 7fb0ab400640 1 -- 172.21.15.84:0/702991053 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0a40108e0 con 0x7fb0b4075310 2024-04-28T12:48:02.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.551+0000 7fb0ab400640 1 -- 172.21.15.84:0/702991053 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4018980 con 0x7fb0b4075310 2024-04-28T12:48:02.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.551+0000 7fb0b3e00640 1 -- 172.21.15.84:0/702991053 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b41a2a60 con 0x7fb0b4075310 2024-04-28T12:48:02.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.551+0000 7fb0b3e00640 1 -- 172.21.15.84:0/702991053 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b41a2f60 con 0x7fb0b4075310 2024-04-28T12:48:02.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.552+0000 7fb0b3e00640 1 -- 172.21.15.84:0/702991053 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b4109420 con 0x7fb0b4075310 2024-04-28T12:48:02.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.554+0000 7fb0ab400640 1 -- 172.21.15.84:0/702991053 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb0a4010430 con 0x7fb0b4075310 2024-04-28T12:48:02.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.555+0000 7fb0ab400640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0980a3bb0 0x7fb0980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:02.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.556+0000 7fb0ab400640 1 -- 172.21.15.84:0/702991053 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb0a40d0230 con 0x7fb0b4075310 2024-04-28T12:48:02.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.556+0000 7fb0abe00640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0980a3bb0 0x7fb0980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:02.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.556+0000 7fb0abe00640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0980a3bb0 0x7fb0980a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb0b419c060 tx=0x7fb09c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:02.556 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.560+0000 7fb0ab400640 1 -- 172.21.15.84:0/702991053 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb0a409ca60 con 0x7fb0b4075310 2024-04-28T12:48:02.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:02 smithi084 ceph-mon[28081]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:02.668 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 12:48:02 smithi084 podman[65468]: 2024-04-28 12:48:02.483908522 +0000 UTC m=+0.457592380 container init a1ee823846e83887fd1c06b05a883b6fa4281c2d430af607a52bbfc423145871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-28T12:48:02.668 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 12:48:02 smithi084 podman[65468]: 2024-04-28 12:48:02.49393621 +0000 UTC m=+0.467619997 container start a1ee823846e83887fd1c06b05a883b6fa4281c2d430af607a52bbfc423145871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T12:48:02.668 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 12:48:02 smithi084 ceph-mds[65507]: -- 172.21.15.84:0/374508561 <== mon.0 v2:172.21.15.84:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55862b5b5a40 con 0x55862b588c00 2024-04-28T12:48:02.668 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 12:48:02 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-d[65483]: starting mds.d at 2024-04-28T12:48:02.668 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 12:48:02 smithi084 bash[65468]: a1ee823846e83887fd1c06b05a883b6fa4281c2d430af607a52bbfc423145871 2024-04-28T12:48:02.668 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 12:48:02 smithi084 systemd[1]: Started Ceph mds.d for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:48:02.826 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.829+0000 7fb0b3e00640 1 -- 172.21.15.84:0/702991053 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0b402dad0 con 0x7fb0b4075310 2024-04-28T12:48:02.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.831+0000 7fb0ab400640 1 -- 172.21.15.84:0/702991053 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7fb0a40a0d60 con 0x7fb0b4075310 2024-04-28T12:48:02.827 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:02.828 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-28T12:48:02.828 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 5 2024-04-28T12:48:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.834+0000 7fb0b3e00640 1 -- 172.21.15.84:0/702991053 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0980a3bb0 msgr2=0x7fb0980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.834+0000 7fb0b3e00640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0980a3bb0 0x7fb0980a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb0b419c060 tx=0x7fb09c005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.834+0000 7fb0b3e00640 1 -- 172.21.15.84:0/702991053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0b4075310 msgr2=0x7fb0b419ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.834+0000 7fb0b3e00640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0b4075310 0x7fb0b419ab50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400e9f0 tx=0x7fb0a400eeb0 comp rx=0 tx=0).stop 2024-04-28T12:48:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.834+0000 7fb0b3e00640 1 -- 172.21.15.84:0/702991053 shutdown_connections 2024-04-28T12:48:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.834+0000 7fb0b3e00640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0980a3bb0 0x7fb0980a6070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.834+0000 7fb0b3e00640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0b410a660 0x7fb0b41a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.834+0000 7fb0b3e00640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0b40768f0 0x7fb0b419b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.834+0000 7fb0b3e00640 1 --2- 172.21.15.84:0/702991053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0b4075310 0x7fb0b419ab50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.834+0000 7fb0b3e00640 1 -- 172.21.15.84:0/702991053 >> 172.21.15.84:0/702991053 conn(0x7fb0b40fdd40 msgr2=0x7fb0b41065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.834+0000 7fb0b3e00640 1 -- 172.21.15.84:0/702991053 shutdown_connections 2024-04-28T12:48:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:02.834+0000 7fb0b3e00640 1 -- 172.21.15.84:0/702991053 wait complete. 2024-04-28T12:48:02.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:02 smithi179 ceph-mon[63029]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:02.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:02 smithi161 ceph-mon[62387]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:03.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:03 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.84:6836/2455857863,v1:172.21.15.84:6837/2455857863] up:boot 2024-04-28T12:48:03.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:03 smithi179 ceph-mon[63029]: fsmap 4 up:standby 2024-04-28T12:48:03.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:03 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-28T12:48:03.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:03 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:03.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:03 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:03.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:03 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:03.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:03 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:03.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:03 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:03.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:03 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:03.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:03 smithi179 ceph-mon[63029]: Deploying daemon mds.e on smithi161 2024-04-28T12:48:03.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:03 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/702991053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:03.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:03 smithi179 ceph-mon[63029]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:03.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:03 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.84:6836/2455857863,v1:172.21.15.84:6837/2455857863] up:boot 2024-04-28T12:48:03.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:03 smithi084 ceph-mon[28081]: fsmap 4 up:standby 2024-04-28T12:48:03.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:03 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-28T12:48:03.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:03 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:03.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:03 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:03.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:03 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:03.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:03 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:03.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:03 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:03.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:03 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:03.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:03 smithi084 ceph-mon[28081]: Deploying daemon mds.e on smithi161 2024-04-28T12:48:03.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:03 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/702991053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:03.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:03 smithi084 ceph-mon[28081]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:03.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:03 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.84:6836/2455857863,v1:172.21.15.84:6837/2455857863] up:boot 2024-04-28T12:48:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:03 smithi161 ceph-mon[62387]: fsmap 4 up:standby 2024-04-28T12:48:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:03 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-28T12:48:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:03 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:03 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:03 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:03 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:03 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:03 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:03 smithi161 ceph-mon[62387]: Deploying daemon mds.e on smithi161 2024-04-28T12:48:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:03 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/702991053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:03 smithi161 ceph-mon[62387]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:04.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:04.993 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 12:48:04 smithi161 systemd[1]: Starting Ceph mds.e for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:48:05.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.061+0000 7f8033e00640 1 -- 172.21.15.84:0/1941559673 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8034103950 msgr2=0x7f8034105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:05.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.061+0000 7f8033e00640 1 --2- 172.21.15.84:0/1941559673 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8034103950 0x7f8034105dc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8020009900 tx=0x7f802002a410 comp rx=0 tx=0).stop 2024-04-28T12:48:05.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.061+0000 7f8033e00640 1 -- 172.21.15.84:0/1941559673 shutdown_connections 2024-04-28T12:48:05.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.061+0000 7f8033e00640 1 --2- 172.21.15.84:0/1941559673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8034106300 0x7f8034106780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.061+0000 7f8033e00640 1 --2- 172.21.15.84:0/1941559673 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8034103950 0x7f8034105dc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.061+0000 7f8033e00640 1 --2- 172.21.15.84:0/1941559673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8034100fe0 0x7f8034103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.061+0000 7f8033e00640 1 -- 172.21.15.84:0/1941559673 >> 172.21.15.84:0/1941559673 conn(0x7f80340fcc00 msgr2=0x7f80340ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:05.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.061+0000 7f8033e00640 1 -- 172.21.15.84:0/1941559673 shutdown_connections 2024-04-28T12:48:05.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.061+0000 7f8033e00640 1 -- 172.21.15.84:0/1941559673 wait complete. 2024-04-28T12:48:05.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8033e00640 1 Processor -- start 2024-04-28T12:48:05.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8033e00640 1 -- start start 2024-04-28T12:48:05.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8033e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8034100fe0 0x7f803419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:05.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8033e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8034103950 0x7f803419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:05.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8033e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8034106300 0x7f80341a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:05.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8033e00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8034114890 con 0x7f8034100fe0 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8033e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8034114710 con 0x7f8034106300 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8033e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f8034114a10 con 0x7f8034103950 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8032a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8034100fe0 0x7f803419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8033400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8034106300 0x7f80341a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8033400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8034106300 0x7f80341a6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:37960/0 (socket says 172.21.15.84:37960) 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8032a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8034100fe0 0x7f803419ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36526/0 (socket says 172.21.15.84:36526) 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8033400640 1 -- 172.21.15.84:0/847493752 learned_addr learned my addr 172.21.15.84:0/847493752 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.062+0000 7f8032000640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8034103950 0x7f803419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.063+0000 7f8033400640 1 -- 172.21.15.84:0/847493752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8034103950 msgr2=0x7f803419f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.063+0000 7f8033400640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8034103950 0x7f803419f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.063+0000 7f8033400640 1 -- 172.21.15.84:0/847493752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8034100fe0 msgr2=0x7f803419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.063+0000 7f8033400640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8034100fe0 0x7f803419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.063+0000 7f8033400640 1 -- 172.21.15.84:0/847493752 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80341a6b90 con 0x7f8034106300 2024-04-28T12:48:05.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.063+0000 7f8032a00640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8034100fe0 0x7f803419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:05.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.063+0000 7f8033400640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8034106300 0x7f80341a6520 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f802c009fd0 tx=0x7f802c00b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:05.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.063+0000 7f802b400640 1 -- 172.21.15.84:0/847493752 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c017070 con 0x7f8034106300 2024-04-28T12:48:05.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.063+0000 7f802b400640 1 -- 172.21.15.84:0/847493752 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f802c003e20 con 0x7f8034106300 2024-04-28T12:48:05.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.063+0000 7f8033e00640 1 -- 172.21.15.84:0/847493752 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80341a6e80 con 0x7f8034106300 2024-04-28T12:48:05.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.063+0000 7f802b400640 1 -- 172.21.15.84:0/847493752 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c010460 con 0x7f8034106300 2024-04-28T12:48:05.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.064+0000 7f8033e00640 1 -- 172.21.15.84:0/847493752 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80341a72f0 con 0x7f8034106300 2024-04-28T12:48:05.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.065+0000 7f8033e00640 1 -- 172.21.15.84:0/847493752 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80340fe6f0 con 0x7f8034106300 2024-04-28T12:48:05.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.066+0000 7f802b400640 1 -- 172.21.15.84:0/847493752 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f802c003970 con 0x7f8034106300 2024-04-28T12:48:05.064 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.068+0000 7f802b400640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f80180a3bb0 0x7f80180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:05.064 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.068+0000 7f802b400640 1 -- 172.21.15.84:0/847493752 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f802c0cfd00 con 0x7f8034106300 2024-04-28T12:48:05.064 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.068+0000 7f8032a00640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f80180a3bb0 0x7f80180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:05.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.069+0000 7f8032a00640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f80180a3bb0 0x7f80180a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8024009fc0 tx=0x7f8024005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:05.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.072+0000 7f802b400640 1 -- 172.21.15.84:0/847493752 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f802c09c490 con 0x7f8034106300 2024-04-28T12:48:05.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.313+0000 7f8033e00640 1 -- 172.21.15.84:0/847493752 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80340635d0 con 0x7f8034106300 2024-04-28T12:48:05.310 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.314+0000 7f802b400640 1 -- 172.21.15.84:0/847493752 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f802c0a0790 con 0x7f8034106300 2024-04-28T12:48:05.310 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:05.311 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-28T12:48:05.311 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 5 2024-04-28T12:48:05.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.316+0000 7f8033e00640 1 -- 172.21.15.84:0/847493752 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f80180a3bb0 msgr2=0x7f80180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:05.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.316+0000 7f8033e00640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f80180a3bb0 0x7f80180a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8024009fc0 tx=0x7f8024005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:05.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.316+0000 7f8033e00640 1 -- 172.21.15.84:0/847493752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8034106300 msgr2=0x7f80341a6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:05.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.316+0000 7f8033e00640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8034106300 0x7f80341a6520 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f802c009fd0 tx=0x7f802c00b260 comp rx=0 tx=0).stop 2024-04-28T12:48:05.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.316+0000 7f8033e00640 1 -- 172.21.15.84:0/847493752 shutdown_connections 2024-04-28T12:48:05.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.316+0000 7f8033e00640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f80180a3bb0 0x7f80180a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.316+0000 7f8033e00640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8034106300 0x7f80341a6520 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.316+0000 7f8033e00640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8034103950 0x7f803419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.317+0000 7f8033e00640 1 --2- 172.21.15.84:0/847493752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8034100fe0 0x7f803419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:05.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.317+0000 7f8033e00640 1 -- 172.21.15.84:0/847493752 >> 172.21.15.84:0/847493752 conn(0x7f80340fcc00 msgr2=0x7f80341018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:05.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.317+0000 7f8033e00640 1 -- 172.21.15.84:0/847493752 shutdown_connections 2024-04-28T12:48:05.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:05.317+0000 7f8033e00640 1 -- 172.21.15.84:0/847493752 wait complete. 2024-04-28T12:48:05.363 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 12:48:05 smithi161 podman[91829]: 2024-04-28 12:48:04.982212291 +0000 UTC m=+0.017614021 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:05.364 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 12:48:05 smithi161 podman[91829]: 2024-04-28 12:48:05.181013819 +0000 UTC m=+0.216415578 container create 3492e64bd3bb539baaef5e44b55daa3af6fc34d3b36aa0a956cc2a85a78e2ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T12:48:05.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:05 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/847493752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:05.666 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:05 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/847493752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:05.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:05 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/847493752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:05.742 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 12:48:05 smithi161 podman[91829]: 2024-04-28 12:48:05.409429953 +0000 UTC m=+0.444831672 container init 3492e64bd3bb539baaef5e44b55daa3af6fc34d3b36aa0a956cc2a85a78e2ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T12:48:05.742 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 12:48:05 smithi161 podman[91829]: 2024-04-28 12:48:05.412226995 +0000 UTC m=+0.447628720 container start 3492e64bd3bb539baaef5e44b55daa3af6fc34d3b36aa0a956cc2a85a78e2ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-04-28T12:48:05.743 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 12:48:05 smithi161 ceph-mds[91848]: -- 172.21.15.161:0/4067658036 <== mon.2 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5649c353fa40 con 0x5649c3513000 2024-04-28T12:48:05.743 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 12:48:05 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-e[91844]: starting mds.e at 2024-04-28T12:48:05.743 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 12:48:05 smithi161 bash[91829]: 3492e64bd3bb539baaef5e44b55daa3af6fc34d3b36aa0a956cc2a85a78e2ba4 2024-04-28T12:48:05.743 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 12:48:05 smithi161 systemd[1]: Started Ceph mds.e for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:48:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:06 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:06 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:06 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:06 smithi179 ceph-mon[63029]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:06 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:06 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:06 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:06 smithi179 ceph-mon[63029]: Deploying daemon mds.f on smithi179 2024-04-28T12:48:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:06 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.161:6834/3348651842,v1:172.21.15.161:6835/3348651842] up:boot 2024-04-28T12:48:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:06 smithi179 ceph-mon[63029]: fsmap 5 up:standby 2024-04-28T12:48:06.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:06 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:06 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:06 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:06 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:06 smithi084 ceph-mon[28081]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:06 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:06 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:06 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:06 smithi084 ceph-mon[28081]: Deploying daemon mds.f on smithi179 2024-04-28T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:06 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.161:6834/3348651842,v1:172.21.15.161:6835/3348651842] up:boot 2024-04-28T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:06 smithi084 ceph-mon[28081]: fsmap 5 up:standby 2024-04-28T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:06 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T12:48:06.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:06 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:06 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:06 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:06 smithi161 ceph-mon[62387]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:06 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:06 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:06 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:06 smithi161 ceph-mon[62387]: Deploying daemon mds.f on smithi179 2024-04-28T12:48:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:06 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.161:6834/3348651842,v1:172.21.15.161:6835/3348651842] up:boot 2024-04-28T12:48:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:06 smithi161 ceph-mon[62387]: fsmap 5 up:standby 2024-04-28T12:48:06.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:06 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T12:48:07.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:07.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.546+0000 7fea4fe00640 1 -- 172.21.15.84:0/2613195361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea50109f80 msgr2=0x7fea5010a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:07.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.546+0000 7fea4fe00640 1 --2- 172.21.15.84:0/2613195361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea50109f80 0x7fea5010a400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009860 tx=0x7fea3c02a340 comp rx=0 tx=0).stop 2024-04-28T12:48:07.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.546+0000 7fea4fe00640 1 -- 172.21.15.84:0/2613195361 shutdown_connections 2024-04-28T12:48:07.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.546+0000 7fea4fe00640 1 --2- 172.21.15.84:0/2613195361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea5010a940 0x7fea501111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.546+0000 7fea4fe00640 1 --2- 172.21.15.84:0/2613195361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea50109f80 0x7fea5010a400 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.546+0000 7fea4fe00640 1 --2- 172.21.15.84:0/2613195361 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea50108d90 0x7fea501091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.546+0000 7fea4fe00640 1 -- 172.21.15.84:0/2613195361 >> 172.21.15.84:0/2613195361 conn(0x7fea50077d30 msgr2=0x7fea5007a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:07.544 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.547+0000 7fea4fe00640 1 -- 172.21.15.84:0/2613195361 shutdown_connections 2024-04-28T12:48:07.544 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.547+0000 7fea4fe00640 1 -- 172.21.15.84:0/2613195361 wait complete. 2024-04-28T12:48:07.545 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.548+0000 7fea4fe00640 1 Processor -- start 2024-04-28T12:48:07.545 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.548+0000 7fea4fe00640 1 -- start start 2024-04-28T12:48:07.545 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.548+0000 7fea4fe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea50108d90 0x7fea5019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.548+0000 7fea4fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea50109f80 0x7fea5019f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.548+0000 7fea4fe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea5010a940 0x7fea501a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.548+0000 7fea4fe00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fea501148a0 con 0x7fea50108d90 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.548+0000 7fea4fe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fea50114720 con 0x7fea50109f80 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.548+0000 7fea4fe00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fea50114a20 con 0x7fea5010a940 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.548+0000 7fea4ea00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea50108d90 0x7fea5019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.549+0000 7fea4ea00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea50108d90 0x7fea5019efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36542/0 (socket says 172.21.15.84:36542) 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.549+0000 7fea4e000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea50109f80 0x7fea5019f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.549+0000 7fea4ea00640 1 -- 172.21.15.84:0/372243175 learned_addr learned my addr 172.21.15.84:0/372243175 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.549+0000 7fea4f400640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea5010a940 0x7fea501a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.549+0000 7fea4ea00640 1 -- 172.21.15.84:0/372243175 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea5010a940 msgr2=0x7fea501a6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.549+0000 7fea4ea00640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea5010a940 0x7fea501a6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.549+0000 7fea4ea00640 1 -- 172.21.15.84:0/372243175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea50109f80 msgr2=0x7fea5019f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:07.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.549+0000 7fea4ea00640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea50109f80 0x7fea5019f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.549+0000 7fea4ea00640 1 -- 172.21.15.84:0/372243175 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea501a6bc0 con 0x7fea50108d90 2024-04-28T12:48:07.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.549+0000 7fea4ea00640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea50108d90 0x7fea5019efa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fea4000e9f0 tx=0x7fea4000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:07.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.550+0000 7fea47400640 1 -- 172.21.15.84:0/372243175 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4000ccc0 con 0x7fea50108d90 2024-04-28T12:48:07.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.550+0000 7fea47400640 1 -- 172.21.15.84:0/372243175 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea400035f0 con 0x7fea50108d90 2024-04-28T12:48:07.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.550+0000 7fea4fe00640 1 -- 172.21.15.84:0/372243175 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea501a6eb0 con 0x7fea50108d90 2024-04-28T12:48:07.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.550+0000 7fea47400640 1 -- 172.21.15.84:0/372243175 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4000ce60 con 0x7fea50108d90 2024-04-28T12:48:07.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.550+0000 7fea4fe00640 1 -- 172.21.15.84:0/372243175 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea501a7310 con 0x7fea50108d90 2024-04-28T12:48:07.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.551+0000 7fea4fe00640 1 -- 172.21.15.84:0/372243175 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea50110180 con 0x7fea50108d90 2024-04-28T12:48:07.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.553+0000 7fea47400640 1 -- 172.21.15.84:0/372243175 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fea40004000 con 0x7fea50108d90 2024-04-28T12:48:07.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.554+0000 7fea47400640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fea340a3bb0 0x7fea340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:07.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.554+0000 7fea47400640 1 -- 172.21.15.84:0/372243175 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fea400d01e0 con 0x7fea50108d90 2024-04-28T12:48:07.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.554+0000 7fea4e000640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fea340a3bb0 0x7fea340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:07.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.555+0000 7fea4e000640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fea340a3bb0 0x7fea340a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fea3c000c00 tx=0x7fea3c030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:07.555 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.559+0000 7fea47400640 1 -- 172.21.15.84:0/372243175 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fea4009cfc0 con 0x7fea50108d90 2024-04-28T12:48:07.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.805+0000 7fea4fe00640 1 -- 172.21.15.84:0/372243175 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea5002dad0 con 0x7fea50108d90 2024-04-28T12:48:07.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.806+0000 7fea47400640 1 -- 172.21.15.84:0/372243175 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7fea4009c8c0 con 0x7fea50108d90 2024-04-28T12:48:07.803 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:07.803 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-28T12:48:07.803 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 6 2024-04-28T12:48:07.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.808+0000 7fea4fe00640 1 -- 172.21.15.84:0/372243175 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fea340a3bb0 msgr2=0x7fea340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:07.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.809+0000 7fea4fe00640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fea340a3bb0 0x7fea340a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fea3c000c00 tx=0x7fea3c030470 comp rx=0 tx=0).stop 2024-04-28T12:48:07.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.809+0000 7fea4fe00640 1 -- 172.21.15.84:0/372243175 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea50108d90 msgr2=0x7fea5019efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:07.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.809+0000 7fea4fe00640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea50108d90 0x7fea5019efa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fea4000e9f0 tx=0x7fea4000eeb0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.809+0000 7fea4fe00640 1 -- 172.21.15.84:0/372243175 shutdown_connections 2024-04-28T12:48:07.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.809+0000 7fea4fe00640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fea340a3bb0 0x7fea340a6070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.809+0000 7fea4fe00640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea5010a940 0x7fea501a6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.809+0000 7fea4fe00640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea50109f80 0x7fea5019f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.809+0000 7fea4fe00640 1 --2- 172.21.15.84:0/372243175 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea50108d90 0x7fea5019efa0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:07.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.809+0000 7fea4fe00640 1 -- 172.21.15.84:0/372243175 >> 172.21.15.84:0/372243175 conn(0x7fea50077d30 msgr2=0x7fea5010d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:07.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.809+0000 7fea4fe00640 1 -- 172.21.15.84:0/372243175 shutdown_connections 2024-04-28T12:48:07.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:07.809+0000 7fea4fe00640 1 -- 172.21.15.84:0/372243175 wait complete. 2024-04-28T12:48:08.131 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 12:48:07 smithi179 systemd[1]: Starting Ceph mds.f for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:48:08.391 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 12:48:08 smithi179 podman[93400]: 2024-04-28 12:48:08.135803071 +0000 UTC m=+0.111680445 container create 0b9c806d90b45c7178b44d99fab87e18329fcf58d7d0673709898fea76e9711b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0) 2024-04-28T12:48:08.392 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 12:48:08 smithi179 podman[93400]: 2024-04-28 12:48:08.04208404 +0000 UTC m=+0.017961430 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:08.392 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 12:48:08 smithi179 podman[93400]: 2024-04-28 12:48:08.396758084 +0000 UTC m=+0.372635456 container init 0b9c806d90b45c7178b44d99fab87e18329fcf58d7d0673709898fea76e9711b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T12:48:08.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:08 smithi179 ceph-mon[63029]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:08.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:08 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/372243175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:08.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:08.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:08.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:08.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:08.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:08.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:08 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:08.665 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 12:48:08 smithi179 podman[93400]: 2024-04-28 12:48:08.399605468 +0000 UTC m=+0.375482849 container start 0b9c806d90b45c7178b44d99fab87e18329fcf58d7d0673709898fea76e9711b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T12:48:08.665 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 12:48:08 smithi179 ceph-mds[93418]: -- 172.21.15.179:0/2993498839 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556ba5b71a40 con 0x556ba5b44800 2024-04-28T12:48:08.665 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 12:48:08 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-f[93414]: starting mds.f at 2024-04-28T12:48:08.665 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 12:48:08 smithi179 bash[93400]: 0b9c806d90b45c7178b44d99fab87e18329fcf58d7d0673709898fea76e9711b 2024-04-28T12:48:08.666 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 12:48:08 smithi179 systemd[1]: Started Ceph mds.f for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:48:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:08 smithi084 ceph-mon[28081]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:08 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/372243175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:08.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:08.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:08.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:08 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:08 smithi161 ceph-mon[62387]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:08 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/372243175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:08 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:09.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:09.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:09 smithi179 ceph-mon[63029]: Deploying daemon mds.c on smithi179 2024-04-28T12:48:09.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:09 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.179:6834/476402611,v1:172.21.15.179:6835/476402611] up:boot 2024-04-28T12:48:09.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:09 smithi179 ceph-mon[63029]: fsmap 6 up:standby 2024-04-28T12:48:09.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:09 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T12:48:09.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:09 smithi084 ceph-mon[28081]: Deploying daemon mds.c on smithi179 2024-04-28T12:48:09.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:09 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.179:6834/476402611,v1:172.21.15.179:6835/476402611] up:boot 2024-04-28T12:48:09.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:09 smithi084 ceph-mon[28081]: fsmap 6 up:standby 2024-04-28T12:48:09.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:09 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T12:48:09.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:09 smithi161 ceph-mon[62387]: Deploying daemon mds.c on smithi179 2024-04-28T12:48:09.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:09 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.179:6834/476402611,v1:172.21.15.179:6835/476402611] up:boot 2024-04-28T12:48:09.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:09 smithi161 ceph-mon[62387]: fsmap 6 up:standby 2024-04-28T12:48:09.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:09 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-28T12:48:10.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.041+0000 7f5bb6200640 1 -- 172.21.15.84:0/539204560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0104990 msgr2=0x7f5bb0104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:10.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.041+0000 7f5bb6200640 1 --2- 172.21.15.84:0/539204560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0104990 0x7f5bb0104db0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0009900 tx=0x7f5ba002a3f0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.041+0000 7f5bb6200640 1 -- 172.21.15.84:0/539204560 shutdown_connections 2024-04-28T12:48:10.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.041+0000 7f5bb6200640 1 --2- 172.21.15.84:0/539204560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0106540 0x7f5bb010cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.041+0000 7f5bb6200640 1 --2- 172.21.15.84:0/539204560 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0105b80 0x7f5bb0106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.041+0000 7f5bb6200640 1 --2- 172.21.15.84:0/539204560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0104990 0x7f5bb0104db0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.041+0000 7f5bb6200640 1 -- 172.21.15.84:0/539204560 >> 172.21.15.84:0/539204560 conn(0x7f5bb00fff00 msgr2=0x7f5bb0102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:10.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.042+0000 7f5bb6200640 1 -- 172.21.15.84:0/539204560 shutdown_connections 2024-04-28T12:48:10.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.042+0000 7f5bb6200640 1 -- 172.21.15.84:0/539204560 wait complete. 2024-04-28T12:48:10.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.043+0000 7f5bb6200640 1 Processor -- start 2024-04-28T12:48:10.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.043+0000 7f5bb6200640 1 -- start start 2024-04-28T12:48:10.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.043+0000 7f5bb6200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0104990 0x7f5bb019ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:10.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.043+0000 7f5bb6200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0105b80 0x7f5bb019b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:10.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.043+0000 7f5bb6200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0106540 0x7f5bb01a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:10.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.043+0000 7f5bb6200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb01104c0 con 0x7f5bb0104990 2024-04-28T12:48:10.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.043+0000 7f5bb6200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb0110340 con 0x7f5bb0105b80 2024-04-28T12:48:10.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.043+0000 7f5bb6200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb0110640 con 0x7f5bb0106540 2024-04-28T12:48:10.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.044+0000 7f5bb4e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0104990 0x7f5bb019ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:10.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.044+0000 7f5bb4e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0104990 0x7f5bb019ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41490/0 (socket says 172.21.15.84:41490) 2024-04-28T12:48:10.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.044+0000 7f5bb4e00640 1 -- 172.21.15.84:0/3718964743 learned_addr learned my addr 172.21.15.84:0/3718964743 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:10.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.044+0000 7f5ba7e00640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0105b80 0x7f5bb019b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:10.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.044+0000 7f5bb5800640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0106540 0x7f5bb01a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:10.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.044+0000 7f5ba7e00640 1 -- 172.21.15.84:0/3718964743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0106540 msgr2=0x7f5bb01a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:10.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.044+0000 7f5ba7e00640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0106540 0x7f5bb01a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.044+0000 7f5ba7e00640 1 -- 172.21.15.84:0/3718964743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0104990 msgr2=0x7f5bb019ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:10.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.044+0000 7f5ba7e00640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0104990 0x7f5bb019ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.044+0000 7f5ba7e00640 1 -- 172.21.15.84:0/3718964743 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb01a27b0 con 0x7f5bb0105b80 2024-04-28T12:48:10.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.044+0000 7f5bb4e00640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0104990 0x7f5bb019ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:10.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.044+0000 7f5ba7e00640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0105b80 0x7f5bb019b0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800e9f0 tx=0x7f5b9800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:10.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.045+0000 7f5bae000640 1 -- 172.21.15.84:0/3718964743 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9800ccc0 con 0x7f5bb0105b80 2024-04-28T12:48:10.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.045+0000 7f5bae000640 1 -- 172.21.15.84:0/3718964743 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b980108e0 con 0x7f5bb0105b80 2024-04-28T12:48:10.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.045+0000 7f5bae000640 1 -- 172.21.15.84:0/3718964743 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b98018880 con 0x7f5bb0105b80 2024-04-28T12:48:10.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.045+0000 7f5bb6200640 1 -- 172.21.15.84:0/3718964743 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb01a2aa0 con 0x7f5bb0105b80 2024-04-28T12:48:10.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.045+0000 7f5bb6200640 1 -- 172.21.15.84:0/3718964743 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb01a3040 con 0x7f5bb0105b80 2024-04-28T12:48:10.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.046+0000 7f5bb6200640 1 -- 172.21.15.84:0/3718964743 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb010bdd0 con 0x7f5bb0105b80 2024-04-28T12:48:10.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.048+0000 7f5bae000640 1 -- 172.21.15.84:0/3718964743 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5b98010430 con 0x7f5bb0105b80 2024-04-28T12:48:10.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.049+0000 7f5bae000640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5b900a3c30 0x7f5b900a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:10.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.049+0000 7f5bae000640 1 -- 172.21.15.84:0/3718964743 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5b980d0280 con 0x7f5bb0105b80 2024-04-28T12:48:10.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.050+0000 7f5bb4e00640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5b900a3c30 0x7f5b900a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:10.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.050+0000 7f5bb4e00640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5b900a3c30 0x7f5b900a60f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0032fd0 tx=0x7f5ba00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:10.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.054+0000 7f5bae000640 1 -- 172.21.15.84:0/3718964743 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5b9809ca10 con 0x7f5bb0105b80 2024-04-28T12:48:10.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.296+0000 7f5bb6200640 1 -- 172.21.15.84:0/3718964743 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bb00635d0 con 0x7f5bb0105b80 2024-04-28T12:48:10.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.297+0000 7f5bae000640 1 -- 172.21.15.84:0/3718964743 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7f5b980a0d10 con 0x7f5bb0105b80 2024-04-28T12:48:10.293 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:10.293 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-28T12:48:10.294 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 7 2024-04-28T12:48:10.296 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.299+0000 7f5bb6200640 1 -- 172.21.15.84:0/3718964743 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5b900a3c30 msgr2=0x7f5b900a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:10.296 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.299+0000 7f5bb6200640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5b900a3c30 0x7f5b900a60f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0032fd0 tx=0x7f5ba00314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.296 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.299+0000 7f5bb6200640 1 -- 172.21.15.84:0/3718964743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0105b80 msgr2=0x7f5bb019b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:10.296 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.299+0000 7f5bb6200640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0105b80 0x7f5bb019b0d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800e9f0 tx=0x7f5b9800eeb0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.296 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.299+0000 7f5bb6200640 1 -- 172.21.15.84:0/3718964743 shutdown_connections 2024-04-28T12:48:10.296 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.299+0000 7f5bb6200640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5b900a3c30 0x7f5b900a60f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.296 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.299+0000 7f5bb6200640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0106540 0x7f5bb01a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.296 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.299+0000 7f5bb6200640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0105b80 0x7f5bb019b0d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.296 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.299+0000 7f5bb6200640 1 --2- 172.21.15.84:0/3718964743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0104990 0x7f5bb019ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:10.296 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.299+0000 7f5bb6200640 1 -- 172.21.15.84:0/3718964743 >> 172.21.15.84:0/3718964743 conn(0x7f5bb00fff00 msgr2=0x7f5bb0108fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:10.297 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.299+0000 7f5bb6200640 1 -- 172.21.15.84:0/3718964743 shutdown_connections 2024-04-28T12:48:10.297 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:10.299+0000 7f5bb6200640 1 -- 172.21.15.84:0/3718964743 wait complete. 2024-04-28T12:48:10.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:10 smithi179 ceph-mon[63029]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:10.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:10 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3718964743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:10.915 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 12:48:10 smithi179 systemd[1]: Starting Ceph mds.c for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:48:10.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:10 smithi084 ceph-mon[28081]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:10.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:10 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3718964743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:10.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:10 smithi161 ceph-mon[62387]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:10.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:10 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3718964743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:11.415 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 12:48:11 smithi179 podman[93706]: 2024-04-28 12:48:11.000959269 +0000 UTC m=+0.017869371 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:11.415 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 12:48:11 smithi179 podman[93706]: 2024-04-28 12:48:11.154879438 +0000 UTC m=+0.171789514 container create bd088b7c273ca273e564b2da6a821f6311b85245e148e140eec00eff0dc66776 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-28T12:48:11.915 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 12:48:11 smithi179 podman[93706]: 2024-04-28 12:48:11.432504987 +0000 UTC m=+0.449415055 container init bd088b7c273ca273e564b2da6a821f6311b85245e148e140eec00eff0dc66776 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-28T12:48:11.915 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 12:48:11 smithi179 podman[93706]: 2024-04-28 12:48:11.434984036 +0000 UTC m=+0.451894106 container start bd088b7c273ca273e564b2da6a821f6311b85245e148e140eec00eff0dc66776 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-28T12:48:11.915 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 12:48:11 smithi179 ceph-mds[93725]: -- 172.21.15.179:0/3118980856 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559eb9f99a40 con 0x559eb9f6d000 2024-04-28T12:48:11.915 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 12:48:11 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-c[93721]: starting mds.c at 2024-04-28T12:48:11.915 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 12:48:11 smithi179 bash[93706]: bd088b7c273ca273e564b2da6a821f6311b85245e148e140eec00eff0dc66776 2024-04-28T12:48:11.915 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 12:48:11 smithi179 systemd[1]: Started Ceph mds.c for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:48:12.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:12.521 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.525+0000 7f7363200640 1 -- 172.21.15.84:0/648534161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f735c10a940 msgr2=0x7f735c1111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:12.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.525+0000 7f7363200640 1 --2- 172.21.15.84:0/648534161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f735c10a940 0x7f735c1111d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7358009ca0 tx=0x7f735802a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.525+0000 7f7363200640 1 -- 172.21.15.84:0/648534161 shutdown_connections 2024-04-28T12:48:12.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.525+0000 7f7363200640 1 --2- 172.21.15.84:0/648534161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f735c10a940 0x7f735c1111d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.525+0000 7f7363200640 1 --2- 172.21.15.84:0/648534161 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f735c109f80 0x7f735c10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.525+0000 7f7363200640 1 --2- 172.21.15.84:0/648534161 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f735c108d90 0x7f735c1091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.525+0000 7f7363200640 1 -- 172.21.15.84:0/648534161 >> 172.21.15.84:0/648534161 conn(0x7f735c077d30 msgr2=0x7f735c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:12.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.525+0000 7f7363200640 1 -- 172.21.15.84:0/648534161 shutdown_connections 2024-04-28T12:48:12.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.525+0000 7f7363200640 1 -- 172.21.15.84:0/648534161 wait complete. 2024-04-28T12:48:12.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.526+0000 7f7363200640 1 Processor -- start 2024-04-28T12:48:12.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.526+0000 7f7363200640 1 -- start start 2024-04-28T12:48:12.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.526+0000 7f7363200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f735c108d90 0x7f735c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.526+0000 7f7363200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f735c109f80 0x7f735c19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.526+0000 7f7363200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f735c10a940 0x7f735c1a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.526+0000 7f7363200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f735c114910 con 0x7f735c10a940 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.526+0000 7f7363200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f735c114790 con 0x7f735c108d90 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.526+0000 7f7363200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f735c114a90 con 0x7f735c109f80 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.527+0000 7f7361e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f735c108d90 0x7f735c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.527+0000 7f7362800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f735c10a940 0x7f735c1a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.527+0000 7f7361e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f735c108d90 0x7f735c19ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:54362/0 (socket says 172.21.15.84:54362) 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.527+0000 7f7361e00640 1 -- 172.21.15.84:0/712707768 learned_addr learned my addr 172.21.15.84:0/712707768 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.527+0000 7f7361400640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f735c109f80 0x7f735c19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.527+0000 7f7361e00640 1 -- 172.21.15.84:0/712707768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f735c109f80 msgr2=0x7f735c19f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.527+0000 7f7361e00640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f735c109f80 0x7f735c19f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.527+0000 7f7361e00640 1 -- 172.21.15.84:0/712707768 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f735c10a940 msgr2=0x7f735c1a64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:12.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.527+0000 7f7361e00640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f735c10a940 0x7f735c1a64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.527+0000 7f7361e00640 1 -- 172.21.15.84:0/712707768 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f735c1a6b50 con 0x7f735c108d90 2024-04-28T12:48:12.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.527+0000 7f7361400640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f735c109f80 0x7f735c19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:12.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.527+0000 7f7361e00640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f735c108d90 0x7f735c19ef30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f735000e9c0 tx=0x7f735000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:12.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.528+0000 7f7356a00640 1 -- 172.21.15.84:0/712707768 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735000cd80 con 0x7f735c108d90 2024-04-28T12:48:12.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.528+0000 7f7356a00640 1 -- 172.21.15.84:0/712707768 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73500035f0 con 0x7f735c108d90 2024-04-28T12:48:12.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.528+0000 7f7363200640 1 -- 172.21.15.84:0/712707768 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f735c1a6e40 con 0x7f735c108d90 2024-04-28T12:48:12.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.528+0000 7f7356a00640 1 -- 172.21.15.84:0/712707768 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7350010690 con 0x7f735c108d90 2024-04-28T12:48:12.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.528+0000 7f7363200640 1 -- 172.21.15.84:0/712707768 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f735c1a7300 con 0x7f735c108d90 2024-04-28T12:48:12.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.529+0000 7f7363200640 1 -- 172.21.15.84:0/712707768 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f735c1100b0 con 0x7f735c108d90 2024-04-28T12:48:12.527 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.531+0000 7f7356a00640 1 -- 172.21.15.84:0/712707768 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f73500037a0 con 0x7f735c108d90 2024-04-28T12:48:12.528 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.532+0000 7f7356a00640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73440a3c30 0x7f73440a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:12.528 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.532+0000 7f7356a00640 1 -- 172.21.15.84:0/712707768 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f73500d02e0 con 0x7f735c108d90 2024-04-28T12:48:12.528 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.532+0000 7f7361400640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73440a3c30 0x7f73440a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:12.529 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.533+0000 7f7361400640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73440a3c30 0x7f73440a60f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f735c1a0440 tx=0x7f734c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:12.532 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.536+0000 7f7356a00640 1 -- 172.21.15.84:0/712707768 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f735009d0c0 con 0x7f735c108d90 2024-04-28T12:48:12.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.781+0000 7f7363200640 1 -- 172.21.15.84:0/712707768 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f735c0635d0 con 0x7f735c108d90 2024-04-28T12:48:12.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.783+0000 7f7356a00640 1 -- 172.21.15.84:0/712707768 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7f735009c9c0 con 0x7f735c108d90 2024-04-28T12:48:12.779 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:12.779 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-28T12:48:12.779 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 8 2024-04-28T12:48:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.785+0000 7f7363200640 1 -- 172.21.15.84:0/712707768 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73440a3c30 msgr2=0x7f73440a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.785+0000 7f7363200640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73440a3c30 0x7f73440a60f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f735c1a0440 tx=0x7f734c005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.785+0000 7f7363200640 1 -- 172.21.15.84:0/712707768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f735c108d90 msgr2=0x7f735c19ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.785+0000 7f7363200640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f735c108d90 0x7f735c19ef30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f735000e9c0 tx=0x7f735000ee80 comp rx=0 tx=0).stop 2024-04-28T12:48:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.785+0000 7f7363200640 1 -- 172.21.15.84:0/712707768 shutdown_connections 2024-04-28T12:48:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.785+0000 7f7363200640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73440a3c30 0x7f73440a60f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.785+0000 7f7363200640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f735c10a940 0x7f735c1a64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.785+0000 7f7363200640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f735c109f80 0x7f735c19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.785+0000 7f7363200640 1 --2- 172.21.15.84:0/712707768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f735c108d90 0x7f735c19ef30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.785+0000 7f7363200640 1 -- 172.21.15.84:0/712707768 >> 172.21.15.84:0/712707768 conn(0x7f735c077d30 msgr2=0x7f735c10d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.785+0000 7f7363200640 1 -- 172.21.15.84:0/712707768 shutdown_connections 2024-04-28T12:48:12.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:12.785+0000 7f7363200640 1 -- 172.21.15.84:0/712707768 wait complete. 2024-04-28T12:48:12.836 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:12 smithi161 ceph-mon[62387]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:12.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:12.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:12.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:12.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:12.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:12.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:12.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:12 smithi161 ceph-mon[62387]: Deploying daemon mds.h on smithi161 2024-04-28T12:48:12.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:12 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.179:6836/2376856149,v1:172.21.15.179:6837/2376856149] up:boot 2024-04-28T12:48:12.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:12 smithi161 ceph-mon[62387]: fsmap 7 up:standby 2024-04-28T12:48:12.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:12 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T12:48:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:12 smithi179 ceph-mon[63029]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:12 smithi179 ceph-mon[63029]: Deploying daemon mds.h on smithi161 2024-04-28T12:48:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:12 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.179:6836/2376856149,v1:172.21.15.179:6837/2376856149] up:boot 2024-04-28T12:48:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:12 smithi179 ceph-mon[63029]: fsmap 7 up:standby 2024-04-28T12:48:12.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:12 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T12:48:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:12 smithi084 ceph-mon[28081]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:12 smithi084 ceph-mon[28081]: Deploying daemon mds.h on smithi161 2024-04-28T12:48:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:12 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.179:6836/2376856149,v1:172.21.15.179:6837/2376856149] up:boot 2024-04-28T12:48:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:12 smithi084 ceph-mon[28081]: fsmap 7 up:standby 2024-04-28T12:48:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:12 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T12:48:13.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:13 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/712707768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:13.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:13 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/712707768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:13.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:13 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/712707768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:13.993 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 12:48:13 smithi161 systemd[1]: Starting Ceph mds.h for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:48:14.309 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 12:48:14 smithi161 podman[92138]: 2024-04-28 12:48:13.95116208 +0000 UTC m=+0.018356730 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:14.309 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 12:48:14 smithi161 podman[92138]: 2024-04-28 12:48:14.052720857 +0000 UTC m=+0.119915489 container create 7c99db4ea81783aec7378c64932e0c642b5fbff25effbcb433dd69fa1ec3c153 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-28T12:48:14.606 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 12:48:14 smithi161 podman[92138]: 2024-04-28 12:48:14.313913741 +0000 UTC m=+0.381108374 container init 7c99db4ea81783aec7378c64932e0c642b5fbff25effbcb433dd69fa1ec3c153 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T12:48:14.606 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 12:48:14 smithi161 podman[92138]: 2024-04-28 12:48:14.317549408 +0000 UTC m=+0.384744044 container start 7c99db4ea81783aec7378c64932e0c642b5fbff25effbcb433dd69fa1ec3c153 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T12:48:14.606 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 12:48:14 smithi161 ceph-mds[92158]: -- 172.21.15.161:0/78337715 <== mon.2 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556c74b91a40 con 0x556c74b65000 2024-04-28T12:48:14.606 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 12:48:14 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h[92154]: starting mds.h at 2024-04-28T12:48:14.606 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 12:48:14 smithi161 bash[92138]: 7c99db4ea81783aec7378c64932e0c642b5fbff25effbcb433dd69fa1ec3c153 2024-04-28T12:48:14.606 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 12:48:14 smithi161 systemd[1]: Started Ceph mds.h for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:48:14.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:14.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:14 smithi084 ceph-mon[28081]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:14.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:14.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:14.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:14.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:14.878 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:14.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:14 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:14.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:14 smithi179 ceph-mon[63029]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:14.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:14 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:14.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:14 smithi161 ceph-mon[62387]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:14.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:14.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:14.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:14.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:14.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:14.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:14 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:15.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.016+0000 7f5c7da00640 1 -- 172.21.15.84:0/2855121827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c78109f80 msgr2=0x7f5c7810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:15.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.016+0000 7f5c7da00640 1 --2- 172.21.15.84:0/2855121827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c78109f80 0x7f5c7810a400 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c60009900 tx=0x7f5c6002a3f0 comp rx=0 tx=0).stop 2024-04-28T12:48:15.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.016+0000 7f5c7da00640 1 -- 172.21.15.84:0/2855121827 shutdown_connections 2024-04-28T12:48:15.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.016+0000 7f5c7da00640 1 --2- 172.21.15.84:0/2855121827 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c7810a940 0x7f5c781111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:15.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.016+0000 7f5c7da00640 1 --2- 172.21.15.84:0/2855121827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c78109f80 0x7f5c7810a400 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:15.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.016+0000 7f5c7da00640 1 --2- 172.21.15.84:0/2855121827 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5c78108d90 0x7f5c781091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:15.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.016+0000 7f5c7da00640 1 -- 172.21.15.84:0/2855121827 >> 172.21.15.84:0/2855121827 conn(0x7f5c78077d30 msgr2=0x7f5c7807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:15.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.016+0000 7f5c7da00640 1 -- 172.21.15.84:0/2855121827 shutdown_connections 2024-04-28T12:48:15.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.016+0000 7f5c7da00640 1 -- 172.21.15.84:0/2855121827 wait complete. 2024-04-28T12:48:15.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.017+0000 7f5c7da00640 1 Processor -- start 2024-04-28T12:48:15.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.018+0000 7f5c7da00640 1 -- start start 2024-04-28T12:48:15.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.018+0000 7f5c7da00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c78108d90 0x7f5c7819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:15.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.018+0000 7f5c7da00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c78109f80 0x7f5c7819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:15.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.018+0000 7f5c7da00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5c7810a940 0x7f5c781a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:15.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.018+0000 7f5c7da00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5c781148a0 con 0x7f5c7810a940 2024-04-28T12:48:15.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.018+0000 7f5c7da00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5c78114720 con 0x7f5c78109f80 2024-04-28T12:48:15.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.018+0000 7f5c7da00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5c78114a20 con 0x7f5c78108d90 2024-04-28T12:48:15.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.018+0000 7f5c77e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c78108d90 0x7f5c7819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:15.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.018+0000 7f5c77e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c78108d90 0x7f5c7819f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:48662/0 (socket says 172.21.15.84:48662) 2024-04-28T12:48:15.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.018+0000 7f5c77e00640 1 -- 172.21.15.84:0/2091618222 learned_addr learned my addr 172.21.15.84:0/2091618222 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:15.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.018+0000 7f5c6f400640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c78109f80 0x7f5c7819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:15.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.018+0000 7f5c7d000640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5c7810a940 0x7f5c781a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:15.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.019+0000 7f5c77e00640 1 -- 172.21.15.84:0/2091618222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c78109f80 msgr2=0x7f5c7819f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:15.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.019+0000 7f5c77e00640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c78109f80 0x7f5c7819f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:15.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.019+0000 7f5c77e00640 1 -- 172.21.15.84:0/2091618222 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5c7810a940 msgr2=0x7f5c781a6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:15.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.019+0000 7f5c77e00640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5c7810a940 0x7f5c781a6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:15.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.019+0000 7f5c77e00640 1 -- 172.21.15.84:0/2091618222 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c781a6bd0 con 0x7f5c78108d90 2024-04-28T12:48:15.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.019+0000 7f5c77e00640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c78108d90 0x7f5c7819f050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800ebc0 tx=0x7f5c6800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:15.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.019+0000 7f5c75600640 1 -- 172.21.15.84:0/2091618222 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c68003b70 con 0x7f5c78108d90 2024-04-28T12:48:15.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.019+0000 7f5c75600640 1 -- 172.21.15.84:0/2091618222 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c68003d10 con 0x7f5c78108d90 2024-04-28T12:48:15.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.019+0000 7f5c75600640 1 -- 172.21.15.84:0/2091618222 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c68019800 con 0x7f5c78108d90 2024-04-28T12:48:15.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.020+0000 7f5c7da00640 1 -- 172.21.15.84:0/2091618222 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c781a6ec0 con 0x7f5c78108d90 2024-04-28T12:48:15.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.020+0000 7f5c7da00640 1 -- 172.21.15.84:0/2091618222 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c781a72d0 con 0x7f5c78108d90 2024-04-28T12:48:15.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.021+0000 7f5c7da00640 1 -- 172.21.15.84:0/2091618222 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c78110180 con 0x7f5c78108d90 2024-04-28T12:48:15.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.023+0000 7f5c75600640 1 -- 172.21.15.84:0/2091618222 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5c68004b60 con 0x7f5c78108d90 2024-04-28T12:48:15.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.024+0000 7f5c75600640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5c5c0a3c30 0x7f5c5c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:15.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.024+0000 7f5c75600640 1 -- 172.21.15.84:0/2091618222 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5c680d09a0 con 0x7f5c78108d90 2024-04-28T12:48:15.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.024+0000 7f5c6f400640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5c5c0a3c30 0x7f5c5c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:15.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.025+0000 7f5c6f400640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5c5c0a3c30 0x7f5c5c0a60f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5c60000c00 tx=0x7f5c600314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:15.022 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.027+0000 7f5c75600640 1 -- 172.21.15.84:0/2091618222 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5c6809d780 con 0x7f5c78108d90 2024-04-28T12:48:15.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.270+0000 7f5c7da00640 1 -- 172.21.15.84:0/2091618222 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c7802dd80 con 0x7f5c78108d90 2024-04-28T12:48:15.267 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.271+0000 7f5c75600640 1 -- 172.21.15.84:0/2091618222 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7f5c6809d080 con 0x7f5c78108d90 2024-04-28T12:48:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:15.267 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-28T12:48:15.267 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 9 2024-04-28T12:48:15.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.274+0000 7f5c7da00640 1 -- 172.21.15.84:0/2091618222 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5c5c0a3c30 msgr2=0x7f5c5c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:15.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.274+0000 7f5c7da00640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5c5c0a3c30 0x7f5c5c0a60f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5c60000c00 tx=0x7f5c600314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:15.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.274+0000 7f5c7da00640 1 -- 172.21.15.84:0/2091618222 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c78108d90 msgr2=0x7f5c7819f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:15.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.274+0000 7f5c7da00640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c78108d90 0x7f5c7819f050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800ebc0 tx=0x7f5c6800c6a0 comp rx=0 tx=0).stop 2024-04-28T12:48:15.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.274+0000 7f5c7da00640 1 -- 172.21.15.84:0/2091618222 shutdown_connections 2024-04-28T12:48:15.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.274+0000 7f5c7da00640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5c5c0a3c30 0x7f5c5c0a60f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:15.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.274+0000 7f5c7da00640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5c7810a940 0x7f5c781a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:15.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.274+0000 7f5c7da00640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c78109f80 0x7f5c7819f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:15.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.274+0000 7f5c7da00640 1 --2- 172.21.15.84:0/2091618222 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5c78108d90 0x7f5c7819f050 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:15.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.274+0000 7f5c7da00640 1 -- 172.21.15.84:0/2091618222 >> 172.21.15.84:0/2091618222 conn(0x7f5c78077d30 msgr2=0x7f5c7810d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:15.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.274+0000 7f5c7da00640 1 -- 172.21.15.84:0/2091618222 shutdown_connections 2024-04-28T12:48:15.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:15.275+0000 7f5c7da00640 1 -- 172.21.15.84:0/2091618222 wait complete. 2024-04-28T12:48:15.885 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:15 smithi179 ceph-mon[63029]: Deploying daemon mds.i on smithi179 2024-04-28T12:48:15.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:15 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] up:boot 2024-04-28T12:48:15.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:15 smithi179 ceph-mon[63029]: fsmap 8 up:standby 2024-04-28T12:48:15.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:15 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T12:48:15.886 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:15 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2091618222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:15.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:15 smithi084 ceph-mon[28081]: Deploying daemon mds.i on smithi179 2024-04-28T12:48:15.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:15 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] up:boot 2024-04-28T12:48:15.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:15 smithi084 ceph-mon[28081]: fsmap 8 up:standby 2024-04-28T12:48:15.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:15 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T12:48:15.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:15 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2091618222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:15.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:15 smithi161 ceph-mon[62387]: Deploying daemon mds.i on smithi179 2024-04-28T12:48:15.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:15 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] up:boot 2024-04-28T12:48:15.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:15 smithi161 ceph-mon[62387]: fsmap 8 up:standby 2024-04-28T12:48:15.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:15 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T12:48:15.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:15 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2091618222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:16.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:16 smithi179 ceph-mon[63029]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:16.915 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 12:48:16 smithi179 systemd[1]: Starting Ceph mds.i for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:48:16.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:16 smithi084 ceph-mon[28081]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:16.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:16 smithi161 ceph-mon[62387]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:17.290 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:17.301 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 12:48:17 smithi179 podman[94014]: 2024-04-28 12:48:16.911283845 +0000 UTC m=+0.018190974 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:17.301 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 12:48:17 smithi179 podman[94014]: 2024-04-28 12:48:17.01958084 +0000 UTC m=+0.126487962 container create b20aa40ebd16506b626bdcd14961a8edc726077bc22f1552de3dc5b00fe07d0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-28T12:48:17.301 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 12:48:17 smithi179 podman[94014]: 2024-04-28 12:48:17.305920806 +0000 UTC m=+0.412827931 container init b20aa40ebd16506b626bdcd14961a8edc726077bc22f1552de3dc5b00fe07d0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T12:48:17.506 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.510+0000 7fd657e00640 1 -- 172.21.15.84:0/1583055037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd658076970 msgr2=0x7fd658074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:17.507 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.510+0000 7fd657e00640 1 --2- 172.21.15.84:0/1583055037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd658076970 0x7fd658074dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd648009900 tx=0x7fd64802a3f0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.507 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.511+0000 7fd657e00640 1 -- 172.21.15.84:0/1583055037 shutdown_connections 2024-04-28T12:48:17.507 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.511+0000 7fd657e00640 1 --2- 172.21.15.84:0/1583055037 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd65810a630 0x7fd65810cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.507 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.511+0000 7fd657e00640 1 --2- 172.21.15.84:0/1583055037 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd658075390 0x7fd658075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.507 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.511+0000 7fd657e00640 1 --2- 172.21.15.84:0/1583055037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd658076970 0x7fd658074dc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.507 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.511+0000 7fd657e00640 1 -- 172.21.15.84:0/1583055037 >> 172.21.15.84:0/1583055037 conn(0x7fd6580fdcc0 msgr2=0x7fd658100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:17.507 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.511+0000 7fd657e00640 1 -- 172.21.15.84:0/1583055037 shutdown_connections 2024-04-28T12:48:17.507 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.511+0000 7fd657e00640 1 -- 172.21.15.84:0/1583055037 wait complete. 2024-04-28T12:48:17.509 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.512+0000 7fd657e00640 1 Processor -- start 2024-04-28T12:48:17.509 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.512+0000 7fd657e00640 1 -- start start 2024-04-28T12:48:17.509 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.512+0000 7fd657e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd658075390 0x7fd65819ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:17.509 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd657e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd658076970 0x7fd65819b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:17.509 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd657e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd65810a630 0x7fd6581a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:17.509 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd657e00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd6581104b0 con 0x7fd658075390 2024-04-28T12:48:17.509 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd657e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd658110330 con 0x7fd65810a630 2024-04-28T12:48:17.509 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd657e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd658110630 con 0x7fd658076970 2024-04-28T12:48:17.509 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd656a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd658075390 0x7fd65819ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:17.509 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd656000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd658076970 0x7fd65819b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:17.510 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd656a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd658075390 0x7fd65819ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41546/0 (socket says 172.21.15.84:41546) 2024-04-28T12:48:17.510 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd656a00640 1 -- 172.21.15.84:0/3235889860 learned_addr learned my addr 172.21.15.84:0/3235889860 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:17.510 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd657400640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd65810a630 0x7fd6581a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:17.510 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd656a00640 1 -- 172.21.15.84:0/3235889860 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd658076970 msgr2=0x7fd65819b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:17.510 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd656a00640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd658076970 0x7fd65819b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.510 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd656a00640 1 -- 172.21.15.84:0/3235889860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd65810a630 msgr2=0x7fd6581a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:17.510 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd656a00640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd65810a630 0x7fd6581a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.510 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd656a00640 1 -- 172.21.15.84:0/3235889860 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6581a2760 con 0x7fd658075390 2024-04-28T12:48:17.510 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd657400640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd65810a630 0x7fd6581a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:17.510 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.513+0000 7fd656000640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd658076970 0x7fd65819b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:17.510 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.514+0000 7fd656a00640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd658075390 0x7fd65819ab40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd648005ab0 tx=0x7fd648003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:17.511 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.515+0000 7fd64f400640 1 -- 172.21.15.84:0/3235889860 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64803f070 con 0x7fd658075390 2024-04-28T12:48:17.511 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.515+0000 7fd64f400640 1 -- 172.21.15.84:0/3235889860 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd648031c50 con 0x7fd658075390 2024-04-28T12:48:17.511 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.515+0000 7fd64f400640 1 -- 172.21.15.84:0/3235889860 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd648035030 con 0x7fd658075390 2024-04-28T12:48:17.511 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.515+0000 7fd657e00640 1 -- 172.21.15.84:0/3235889860 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6581a29f0 con 0x7fd658075390 2024-04-28T12:48:17.511 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.515+0000 7fd657e00640 1 -- 172.21.15.84:0/3235889860 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6581a2f30 con 0x7fd658075390 2024-04-28T12:48:17.512 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.516+0000 7fd657e00640 1 -- 172.21.15.84:0/3235889860 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd658109360 con 0x7fd658075390 2024-04-28T12:48:17.513 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.518+0000 7fd64f400640 1 -- 172.21.15.84:0/3235889860 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd648042070 con 0x7fd658075390 2024-04-28T12:48:17.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.519+0000 7fd64f400640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd63c0a3c30 0x7fd63c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:17.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.519+0000 7fd64f400640 1 -- 172.21.15.84:0/3235889860 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd6480f0740 con 0x7fd658075390 2024-04-28T12:48:17.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.520+0000 7fd656000640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd63c0a3c30 0x7fd63c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:17.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.520+0000 7fd656000640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd63c0a3c30 0x7fd63c0a60f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd65819c050 tx=0x7fd640005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:17.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.527+0000 7fd64f400640 1 -- 172.21.15.84:0/3235889860 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd6480bd520 con 0x7fd658075390 2024-04-28T12:48:17.666 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 12:48:17 smithi179 podman[94014]: 2024-04-28 12:48:17.308700842 +0000 UTC m=+0.415607967 container start b20aa40ebd16506b626bdcd14961a8edc726077bc22f1552de3dc5b00fe07d0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T12:48:17.666 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 12:48:17 smithi179 ceph-mds[94033]: -- 172.21.15.179:0/1561812108 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc0c487a40 con 0x55dc0c45ac00 2024-04-28T12:48:17.666 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 12:48:17 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-i[94029]: starting mds.i at 2024-04-28T12:48:17.666 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 12:48:17 smithi179 bash[94014]: b20aa40ebd16506b626bdcd14961a8edc726077bc22f1552de3dc5b00fe07d0f 2024-04-28T12:48:17.666 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 12:48:17 smithi179 systemd[1]: Started Ceph mds.i for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:48:17.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.774+0000 7fd657e00640 1 -- 172.21.15.84:0/3235889860 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd65802dad0 con 0x7fd658075390 2024-04-28T12:48:17.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.776+0000 7fd64f400640 1 -- 172.21.15.84:0/3235889860 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7fd6480bce20 con 0x7fd658075390 2024-04-28T12:48:17.771 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:17.772 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-28T12:48:17.772 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 10 2024-04-28T12:48:17.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.779+0000 7fd657e00640 1 -- 172.21.15.84:0/3235889860 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd63c0a3c30 msgr2=0x7fd63c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:17.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.779+0000 7fd657e00640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd63c0a3c30 0x7fd63c0a60f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd65819c050 tx=0x7fd640005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:17.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.779+0000 7fd657e00640 1 -- 172.21.15.84:0/3235889860 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd658075390 msgr2=0x7fd65819ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:17.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.779+0000 7fd657e00640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd658075390 0x7fd65819ab40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd648005ab0 tx=0x7fd648003b90 comp rx=0 tx=0).stop 2024-04-28T12:48:17.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.779+0000 7fd657e00640 1 -- 172.21.15.84:0/3235889860 shutdown_connections 2024-04-28T12:48:17.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.779+0000 7fd657e00640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd63c0a3c30 0x7fd63c0a60f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.779+0000 7fd657e00640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd65810a630 0x7fd6581a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.779+0000 7fd657e00640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd658076970 0x7fd65819b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.779+0000 7fd657e00640 1 --2- 172.21.15.84:0/3235889860 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd658075390 0x7fd65819ab40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:17.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.779+0000 7fd657e00640 1 -- 172.21.15.84:0/3235889860 >> 172.21.15.84:0/3235889860 conn(0x7fd6580fdcc0 msgr2=0x7fd658106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:17.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.780+0000 7fd657e00640 1 -- 172.21.15.84:0/3235889860 shutdown_connections 2024-04-28T12:48:17.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:17.780+0000 7fd657e00640 1 -- 172.21.15.84:0/3235889860 wait complete. 2024-04-28T12:48:18.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:18.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:18.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:18.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:18.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:18.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:18.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:18 smithi179 ceph-mon[63029]: Deploying daemon mds.g on smithi084 2024-04-28T12:48:18.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:18 smithi179 ceph-mon[63029]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:18.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:18 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.179:6838/3879704230,v1:172.21.15.179:6839/3879704230] up:boot 2024-04-28T12:48:18.726 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:18 smithi179 ceph-mon[63029]: fsmap 9 up:standby 2024-04-28T12:48:18.727 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:18 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T12:48:18.727 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:18 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3235889860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:18.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:18.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:18.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:18.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:18.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:18.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:18.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:18 smithi161 ceph-mon[62387]: Deploying daemon mds.g on smithi084 2024-04-28T12:48:18.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:18 smithi161 ceph-mon[62387]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:18.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:18 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.179:6838/3879704230,v1:172.21.15.179:6839/3879704230] up:boot 2024-04-28T12:48:18.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:18 smithi161 ceph-mon[62387]: fsmap 9 up:standby 2024-04-28T12:48:18.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:18 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T12:48:18.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:18 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3235889860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:18.799 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:18.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:18.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:18.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:18.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:18.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:18.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:18 smithi084 ceph-mon[28081]: Deploying daemon mds.g on smithi084 2024-04-28T12:48:18.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:18 smithi084 ceph-mon[28081]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:18.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:18 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.179:6838/3879704230,v1:172.21.15.179:6839/3879704230] up:boot 2024-04-28T12:48:18.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:18 smithi084 ceph-mon[28081]: fsmap 9 up:standby 2024-04-28T12:48:18.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:18 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T12:48:18.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:18 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3235889860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:19.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:20.008 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.013+0000 7fd555400640 1 -- 172.21.15.84:0/1908802864 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5501027b0 msgr2=0x7fd550102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:20.009 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.013+0000 7fd555400640 1 --2- 172.21.15.84:0/1908802864 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5501027b0 0x7fd550102bd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd540009860 tx=0x7fd54002a340 comp rx=0 tx=0).stop 2024-04-28T12:48:20.009 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.013+0000 7fd555400640 1 -- 172.21.15.84:0/1908802864 shutdown_connections 2024-04-28T12:48:20.009 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.013+0000 7fd555400640 1 --2- 172.21.15.84:0/1908802864 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd550104360 0x7fd55010abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:20.009 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.013+0000 7fd555400640 1 --2- 172.21.15.84:0/1908802864 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5501039a0 0x7fd550103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:20.009 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.013+0000 7fd555400640 1 --2- 172.21.15.84:0/1908802864 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5501027b0 0x7fd550102bd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:20.009 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.013+0000 7fd555400640 1 -- 172.21.15.84:0/1908802864 >> 172.21.15.84:0/1908802864 conn(0x7fd5500fdd40 msgr2=0x7fd550100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:20.009 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.013+0000 7fd555400640 1 -- 172.21.15.84:0/1908802864 shutdown_connections 2024-04-28T12:48:20.009 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.013+0000 7fd555400640 1 -- 172.21.15.84:0/1908802864 wait complete. 2024-04-28T12:48:20.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.014+0000 7fd555400640 1 Processor -- start 2024-04-28T12:48:20.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.014+0000 7fd555400640 1 -- start start 2024-04-28T12:48:20.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.014+0000 7fd555400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5501027b0 0x7fd55019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.014+0000 7fd555400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5501039a0 0x7fd55019b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.014+0000 7fd555400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd550104360 0x7fd5501a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd555400640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd55010e2e0 con 0x7fd550104360 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd555400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd55010e160 con 0x7fd5501027b0 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd555400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd55010e460 con 0x7fd5501039a0 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd554a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd550104360 0x7fd5501a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd54f400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5501039a0 0x7fd55019b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd554a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd550104360 0x7fd5501a2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41458/0 (socket says 172.21.15.84:41458) 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd54f400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5501039a0 0x7fd55019b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:52934/0 (socket says 172.21.15.84:52934) 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd554a00640 1 -- 172.21.15.84:0/1702323656 learned_addr learned my addr 172.21.15.84:0/1702323656 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd54fe00640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5501027b0 0x7fd55019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd54f400640 1 -- 172.21.15.84:0/1702323656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5501027b0 msgr2=0x7fd55019aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd54f400640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5501027b0 0x7fd55019aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd54f400640 1 -- 172.21.15.84:0/1702323656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd550104360 msgr2=0x7fd5501a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:20.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd54f400640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd550104360 0x7fd5501a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:20.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.015+0000 7fd54f400640 1 -- 172.21.15.84:0/1702323656 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5501a27c0 con 0x7fd5501039a0 2024-04-28T12:48:20.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.016+0000 7fd54f400640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5501039a0 0x7fd55019b0e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd538005fd0 tx=0x7fd53800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:20.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.016+0000 7fd54cc00640 1 -- 172.21.15.84:0/1702323656 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53800cd00 con 0x7fd5501039a0 2024-04-28T12:48:20.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.016+0000 7fd54cc00640 1 -- 172.21.15.84:0/1702323656 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5380044d0 con 0x7fd5501039a0 2024-04-28T12:48:20.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.016+0000 7fd555400640 1 -- 172.21.15.84:0/1702323656 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5501a2ab0 con 0x7fd5501039a0 2024-04-28T12:48:20.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.016+0000 7fd54cc00640 1 -- 172.21.15.84:0/1702323656 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd538010430 con 0x7fd5501039a0 2024-04-28T12:48:20.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.016+0000 7fd555400640 1 -- 172.21.15.84:0/1702323656 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5501a3050 con 0x7fd5501039a0 2024-04-28T12:48:20.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.018+0000 7fd555400640 1 -- 172.21.15.84:0/1702323656 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5501a3280 con 0x7fd5501039a0 2024-04-28T12:48:20.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.020+0000 7fd54cc00640 1 -- 172.21.15.84:0/1702323656 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd5380035f0 con 0x7fd5501039a0 2024-04-28T12:48:20.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.021+0000 7fd54cc00640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5340a3c30 0x7fd5340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:20.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.022+0000 7fd54cc00640 1 -- 172.21.15.84:0/1702323656 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd5380d01a0 con 0x7fd5501039a0 2024-04-28T12:48:20.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.022+0000 7fd54fe00640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5340a3c30 0x7fd5340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:20.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.022+0000 7fd54fe00640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5340a3c30 0x7fd5340a60f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd54002a930 tx=0x7fd540030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:20.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.026+0000 7fd54cc00640 1 -- 172.21.15.84:0/1702323656 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd538009510 con 0x7fd5501039a0 2024-04-28T12:48:20.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.290+0000 7fd555400640 1 -- 172.21.15.84:0/1702323656 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5501a3450 con 0x7fd5501039a0 2024-04-28T12:48:20.286 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.291+0000 7fd54cc00640 1 -- 172.21.15.84:0/1702323656 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7fd53809cf80 con 0x7fd5501039a0 2024-04-28T12:48:20.287 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:20.287 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-28T12:48:20.287 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 10 2024-04-28T12:48:20.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.294+0000 7fd555400640 1 -- 172.21.15.84:0/1702323656 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5340a3c30 msgr2=0x7fd5340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:20.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.294+0000 7fd555400640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5340a3c30 0x7fd5340a60f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd54002a930 tx=0x7fd540030470 comp rx=0 tx=0).stop 2024-04-28T12:48:20.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.294+0000 7fd555400640 1 -- 172.21.15.84:0/1702323656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5501039a0 msgr2=0x7fd55019b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:20.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.294+0000 7fd555400640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5501039a0 0x7fd55019b0e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd538005fd0 tx=0x7fd53800ee50 comp rx=0 tx=0).stop 2024-04-28T12:48:20.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.294+0000 7fd555400640 1 -- 172.21.15.84:0/1702323656 shutdown_connections 2024-04-28T12:48:20.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.294+0000 7fd555400640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5340a3c30 0x7fd5340a60f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:20.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.294+0000 7fd555400640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd550104360 0x7fd5501a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:20.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.294+0000 7fd555400640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5501039a0 0x7fd55019b0e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:20.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.294+0000 7fd555400640 1 --2- 172.21.15.84:0/1702323656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5501027b0 0x7fd55019aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:20.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.294+0000 7fd555400640 1 -- 172.21.15.84:0/1702323656 >> 172.21.15.84:0/1702323656 conn(0x7fd5500fdd40 msgr2=0x7fd550074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:20.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.294+0000 7fd555400640 1 -- 172.21.15.84:0/1702323656 shutdown_connections 2024-04-28T12:48:20.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:20.294+0000 7fd555400640 1 -- 172.21.15.84:0/1702323656 wait complete. 2024-04-28T12:48:20.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:20 smithi179 ceph-mon[63029]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:20 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1702323656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:20.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:20 smithi084 ceph-mon[28081]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:20.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:20 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1702323656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:20.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:20 smithi161 ceph-mon[62387]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:20.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:20 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1702323656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:21.356 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 12:48:21 smithi084 podman[66114]: 2024-04-28 12:48:21.167652562 +0000 UTC m=+0.289638353 container create 9db94e8bc7f0ace0af82a8e93b939b2e2a7d9cbf601fe719dc87b5785b11107b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-28T12:48:21.917 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 12:48:21 smithi084 podman[66114]: 2024-04-28 12:48:21.504832412 +0000 UTC m=+0.626818202 container init 9db94e8bc7f0ace0af82a8e93b939b2e2a7d9cbf601fe719dc87b5785b11107b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T12:48:21.917 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 12:48:21 smithi084 podman[66114]: 2024-04-28 12:48:21.514693408 +0000 UTC m=+0.636679199 container start 9db94e8bc7f0ace0af82a8e93b939b2e2a7d9cbf601fe719dc87b5785b11107b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True) 2024-04-28T12:48:21.917 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 12:48:21 smithi084 ceph-mds[66132]: -- 172.21.15.84:0/2479312395 <== mon.2 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5640ff323a40 con 0x5640ff2f6c00 2024-04-28T12:48:21.917 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 12:48:21 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g[66128]: starting mds.g at 2024-04-28T12:48:21.917 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 12:48:21 smithi084 bash[66114]: 9db94e8bc7f0ace0af82a8e93b939b2e2a7d9cbf601fe719dc87b5785b11107b 2024-04-28T12:48:21.917 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 12:48:21 smithi084 systemd[1]: Started Ceph mds.g for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:48:22.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:22.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.519+0000 7f91f9200640 1 -- 172.21.15.84:0/2462523339 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f4076910 msgr2=0x7f91f4074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:22.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.519+0000 7f91f9200640 1 --2- 172.21.15.84:0/2462523339 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f4076910 0x7f91f4074d60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f91e4009ca0 tx=0x7f91e402a950 comp rx=0 tx=0).stop 2024-04-28T12:48:22.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.520+0000 7f91f9200640 1 -- 172.21.15.84:0/2462523339 shutdown_connections 2024-04-28T12:48:22.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.520+0000 7f91f9200640 1 --2- 172.21.15.84:0/2462523339 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f91f410a600 0x7f91f410caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.520+0000 7f91f9200640 1 --2- 172.21.15.84:0/2462523339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91f4075330 0x7f91f4075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.520+0000 7f91f9200640 1 --2- 172.21.15.84:0/2462523339 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f4076910 0x7f91f4074d60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.520+0000 7f91f9200640 1 -- 172.21.15.84:0/2462523339 >> 172.21.15.84:0/2462523339 conn(0x7f91f40fdcc0 msgr2=0x7f91f4100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:22.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.520+0000 7f91f9200640 1 -- 172.21.15.84:0/2462523339 shutdown_connections 2024-04-28T12:48:22.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.520+0000 7f91f9200640 1 -- 172.21.15.84:0/2462523339 wait complete. 2024-04-28T12:48:22.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.521+0000 7f91f9200640 1 Processor -- start 2024-04-28T12:48:22.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.521+0000 7f91f9200640 1 -- start start 2024-04-28T12:48:22.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.521+0000 7f91f9200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91f4075330 0x7f91f419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:22.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.521+0000 7f91f9200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f4076910 0x7f91f419b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:22.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.521+0000 7f91f9200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f91f410a600 0x7f91f41a20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.521+0000 7f91f9200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f91f4110470 con 0x7f91f4076910 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.521+0000 7f91f9200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f91f41102f0 con 0x7f91f410a600 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.521+0000 7f91f9200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f91f41105f0 con 0x7f91f4075330 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.522+0000 7f91eaa00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f4076910 0x7f91f419b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.522+0000 7f91f3400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91f4075330 0x7f91f419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.522+0000 7f91f3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f91f410a600 0x7f91f41a20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.522+0000 7f91f3400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91f4075330 0x7f91f419ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:52980/0 (socket says 172.21.15.84:52980) 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.522+0000 7f91eaa00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f4076910 0x7f91f419b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41492/0 (socket says 172.21.15.84:41492) 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.522+0000 7f91f3400640 1 -- 172.21.15.84:0/1401555230 learned_addr learned my addr 172.21.15.84:0/1401555230 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.522+0000 7f91f3e00640 1 -- 172.21.15.84:0/1401555230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91f4075330 msgr2=0x7f91f419ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.522+0000 7f91f3e00640 1 --2- 172.21.15.84:0/1401555230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91f4075330 0x7f91f419ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.522+0000 7f91f3e00640 1 -- 172.21.15.84:0/1401555230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f4076910 msgr2=0x7f91f419b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.522+0000 7f91f3e00640 1 --2- 172.21.15.84:0/1401555230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f4076910 0x7f91f419b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.522+0000 7f91f3e00640 1 -- 172.21.15.84:0/1401555230 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f41a2750 con 0x7f91f410a600 2024-04-28T12:48:22.519 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.522+0000 7f91f3e00640 1 --2- 172.21.15.84:0/1401555230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f91f410a600 0x7f91f41a20e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00e970 tx=0x7f91ec00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:22.519 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.523+0000 7f91f0c00640 1 -- 172.21.15.84:0/1401555230 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec00ccc0 con 0x7f91f410a600 2024-04-28T12:48:22.519 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.523+0000 7f91f9200640 1 -- 172.21.15.84:0/1401555230 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f41a2a40 con 0x7f91f410a600 2024-04-28T12:48:22.520 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.523+0000 7f91f0c00640 1 -- 172.21.15.84:0/1401555230 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91ec005020 con 0x7f91f410a600 2024-04-28T12:48:22.520 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.523+0000 7f91f0c00640 1 -- 172.21.15.84:0/1401555230 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec004a00 con 0x7f91f410a600 2024-04-28T12:48:22.520 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.523+0000 7f91f9200640 1 -- 172.21.15.84:0/1401555230 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f41a2f90 con 0x7f91f410a600 2024-04-28T12:48:22.520 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.524+0000 7f91f9200640 1 -- 172.21.15.84:0/1401555230 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91f41093c0 con 0x7f91f410a600 2024-04-28T12:48:22.521 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.526+0000 7f91f0c00640 1 -- 172.21.15.84:0/1401555230 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f91ec004ba0 con 0x7f91f410a600 2024-04-28T12:48:22.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.527+0000 7f91f0c00640 1 --2- 172.21.15.84:0/1401555230 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f91d80a3c30 0x7f91d80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:22.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.528+0000 7f91f0c00640 1 -- 172.21.15.84:0/1401555230 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f91ec0cfe70 con 0x7f91f410a600 2024-04-28T12:48:22.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.528+0000 7f91f3400640 1 --2- 172.21.15.84:0/1401555230 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f91d80a3c30 0x7f91d80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:22.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.528+0000 7f91f3400640 1 --2- 172.21.15.84:0/1401555230 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f91d80a3c30 0x7f91d80a60f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f91e4005e30 tx=0x7f91e402f870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:22.527 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.532+0000 7f91f0c00640 1 -- 172.21.15.84:0/1401555230 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f91ec09cc50 con 0x7f91f410a600 2024-04-28T12:48:22.653 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:22 smithi084 ceph-mon[28081]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:22.654 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:22 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] up:boot 2024-04-28T12:48:22.654 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:22 smithi084 ceph-mon[28081]: fsmap 10 up:standby 2024-04-28T12:48:22.654 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:22 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-28T12:48:22.654 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:22 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:22.654 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:22 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:22.654 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:22 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:22.654 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:22 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:22.654 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:22 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:22.654 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:22 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:22.792 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.796+0000 7f91f9200640 1 -- 172.21.15.84:0/1401555230 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91f402dad0 con 0x7f91f410a600 2024-04-28T12:48:22.793 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.797+0000 7f91f0c00640 1 -- 172.21.15.84:0/1401555230 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8839 (secure 0 0 0) 0x7f91ec09c550 con 0x7f91f410a600 2024-04-28T12:48:22.793 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:22.793 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24562,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-28T12:48:22.793 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 11 2024-04-28T12:48:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.800+0000 7f91f9200640 1 -- 172.21.15.84:0/1401555230 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f91d80a3c30 msgr2=0x7f91d80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.800+0000 7f91f9200640 1 --2- 172.21.15.84:0/1401555230 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f91d80a3c30 0x7f91d80a60f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f91e4005e30 tx=0x7f91e402f870 comp rx=0 tx=0).stop 2024-04-28T12:48:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.800+0000 7f91f9200640 1 -- 172.21.15.84:0/1401555230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f91f410a600 msgr2=0x7f91f41a20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.800+0000 7f91f9200640 1 --2- 172.21.15.84:0/1401555230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f91f410a600 0x7f91f41a20e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00e970 tx=0x7f91ec00ee30 comp rx=0 tx=0).stop 2024-04-28T12:48:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.800+0000 7f91f9200640 1 -- 172.21.15.84:0/1401555230 shutdown_connections 2024-04-28T12:48:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.800+0000 7f91f9200640 1 --2- 172.21.15.84:0/1401555230 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f91d80a3c30 0x7f91d80a60f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.800+0000 7f91f9200640 1 --2- 172.21.15.84:0/1401555230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f91f410a600 0x7f91f41a20e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.800+0000 7f91f9200640 1 --2- 172.21.15.84:0/1401555230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91f4076910 0x7f91f419b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.800+0000 7f91f9200640 1 --2- 172.21.15.84:0/1401555230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91f4075330 0x7f91f419ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.800+0000 7f91f9200640 1 -- 172.21.15.84:0/1401555230 >> 172.21.15.84:0/1401555230 conn(0x7f91f40fdcc0 msgr2=0x7f91f4106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.800+0000 7f91f9200640 1 -- 172.21.15.84:0/1401555230 shutdown_connections 2024-04-28T12:48:22.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:22.800+0000 7f91f9200640 1 -- 172.21.15.84:0/1401555230 wait complete. 2024-04-28T12:48:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:22 smithi179 ceph-mon[63029]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:22 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] up:boot 2024-04-28T12:48:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:22 smithi179 ceph-mon[63029]: fsmap 10 up:standby 2024-04-28T12:48:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:22 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-28T12:48:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:22 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:22 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:22 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:22 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:22 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:22 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:22.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:22 smithi161 ceph-mon[62387]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:22.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:22 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] up:boot 2024-04-28T12:48:22.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:22 smithi161 ceph-mon[62387]: fsmap 10 up:standby 2024-04-28T12:48:22.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:22 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-28T12:48:22.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:22 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:22.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:22 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:22.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:22 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:22.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:22 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:22.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:22 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:22.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:22 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:23.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:23 smithi179 ceph-mon[63029]: Deploying daemon mds.a on smithi084 2024-04-28T12:48:23.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:23 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1401555230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:23.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:23 smithi084 ceph-mon[28081]: Deploying daemon mds.a on smithi084 2024-04-28T12:48:23.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:23 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1401555230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:23.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:23 smithi161 ceph-mon[62387]: Deploying daemon mds.a on smithi084 2024-04-28T12:48:23.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:23 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1401555230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:24.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:24.886 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:24 smithi084 ceph-mon[28081]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:24.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:24 smithi161 ceph-mon[62387]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:25.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.020+0000 7f2954200640 1 -- 172.21.15.84:0/1079387432 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f294c105d80 msgr2=0x7f294c106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:25.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.020+0000 7f2954200640 1 --2- 172.21.15.84:0/1079387432 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f294c105d80 0x7f294c106200 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2938009860 tx=0x7f293802a340 comp rx=0 tx=0).stop 2024-04-28T12:48:25.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.020+0000 7f2954200640 1 -- 172.21.15.84:0/1079387432 shutdown_connections 2024-04-28T12:48:25.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.020+0000 7f2954200640 1 --2- 172.21.15.84:0/1079387432 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f294c106740 0x7f294c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:25.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.020+0000 7f2954200640 1 --2- 172.21.15.84:0/1079387432 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f294c105d80 0x7f294c106200 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:25.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.020+0000 7f2954200640 1 --2- 172.21.15.84:0/1079387432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f294c104b90 0x7f294c104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:25.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.021+0000 7f2954200640 1 -- 172.21.15.84:0/1079387432 >> 172.21.15.84:0/1079387432 conn(0x7f294c100120 msgr2=0x7f294c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:25.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.021+0000 7f2954200640 1 -- 172.21.15.84:0/1079387432 shutdown_connections 2024-04-28T12:48:25.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.021+0000 7f2954200640 1 -- 172.21.15.84:0/1079387432 wait complete. 2024-04-28T12:48:25.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.022+0000 7f2954200640 1 Processor -- start 2024-04-28T12:48:25.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.022+0000 7f2954200640 1 -- start start 2024-04-28T12:48:25.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.022+0000 7f2954200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f294c104b90 0x7f294c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:25.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.022+0000 7f2954200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f294c105d80 0x7f294c19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:25.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.022+0000 7f2954200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f294c106740 0x7f294c1a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:25.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.022+0000 7f2954200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f294c1106c0 con 0x7f294c105d80 2024-04-28T12:48:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.022+0000 7f2954200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f294c110540 con 0x7f294c106740 2024-04-28T12:48:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.022+0000 7f2954200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f294c110840 con 0x7f294c104b90 2024-04-28T12:48:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.022+0000 7f2951800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f294c104b90 0x7f294c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.022+0000 7f2950e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f294c105d80 0x7f294c19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.022+0000 7f2951800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f294c104b90 0x7f294c19adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:53002/0 (socket says 172.21.15.84:53002) 2024-04-28T12:48:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.022+0000 7f2951800640 1 -- 172.21.15.84:0/2186076031 learned_addr learned my addr 172.21.15.84:0/2186076031 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.023+0000 7f2952200640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f294c106740 0x7f294c1a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.023+0000 7f2951800640 1 -- 172.21.15.84:0/2186076031 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f294c106740 msgr2=0x7f294c1a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.023+0000 7f2951800640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f294c106740 0x7f294c1a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.023+0000 7f2951800640 1 -- 172.21.15.84:0/2186076031 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f294c105d80 msgr2=0x7f294c19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.023+0000 7f2951800640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f294c105d80 0x7f294c19b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.023+0000 7f2951800640 1 -- 172.21.15.84:0/2186076031 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c1a29d0 con 0x7f294c104b90 2024-04-28T12:48:25.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.023+0000 7f2951800640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f294c104b90 0x7f294c19adc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f293c00e9c0 tx=0x7f293c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:25.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.024+0000 7f294a000640 1 -- 172.21.15.84:0/2186076031 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c00cd80 con 0x7f294c104b90 2024-04-28T12:48:25.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.024+0000 7f294a000640 1 -- 172.21.15.84:0/2186076031 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f293c0035f0 con 0x7f294c104b90 2024-04-28T12:48:25.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.024+0000 7f2954200640 1 -- 172.21.15.84:0/2186076031 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294c1a2cc0 con 0x7f294c104b90 2024-04-28T12:48:25.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.024+0000 7f294a000640 1 -- 172.21.15.84:0/2186076031 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c010690 con 0x7f294c104b90 2024-04-28T12:48:25.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.024+0000 7f2954200640 1 -- 172.21.15.84:0/2186076031 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294c1a31c0 con 0x7f294c104b90 2024-04-28T12:48:25.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.025+0000 7f2954200640 1 -- 172.21.15.84:0/2186076031 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f294c10bf30 con 0x7f294c104b90 2024-04-28T12:48:25.022 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.027+0000 7f294a000640 1 -- 172.21.15.84:0/2186076031 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f293c004000 con 0x7f294c104b90 2024-04-28T12:48:25.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.028+0000 7f294a000640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f29300a3c30 0x7f29300a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:25.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.028+0000 7f294a000640 1 -- 172.21.15.84:0/2186076031 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f293c0d05c0 con 0x7f294c104b90 2024-04-28T12:48:25.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.029+0000 7f2950e00640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f29300a3c30 0x7f29300a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:25.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.029+0000 7f2950e00640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f29300a3c30 0x7f29300a60f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f293802a930 tx=0x7f2938030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:25.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.031+0000 7f294a000640 1 -- 172.21.15.84:0/2186076031 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f293c09d3a0 con 0x7f294c104b90 2024-04-28T12:48:25.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:24 smithi179 ceph-mon[63029]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:25.278 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.283+0000 7f2954200640 1 -- 172.21.15.84:0/2186076031 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f294c02dad0 con 0x7f294c104b90 2024-04-28T12:48:25.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.284+0000 7f294a000640 1 -- 172.21.15.84:0/2186076031 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8839 (secure 0 0 0) 0x7f293c09cca0 con 0x7f294c104b90 2024-04-28T12:48:25.279 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:25.280 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24562,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-28T12:48:25.280 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 11 2024-04-28T12:48:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.287+0000 7f2954200640 1 -- 172.21.15.84:0/2186076031 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f29300a3c30 msgr2=0x7f29300a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.287+0000 7f2954200640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f29300a3c30 0x7f29300a60f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f293802a930 tx=0x7f2938030450 comp rx=0 tx=0).stop 2024-04-28T12:48:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.287+0000 7f2954200640 1 -- 172.21.15.84:0/2186076031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f294c104b90 msgr2=0x7f294c19adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.287+0000 7f2954200640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f294c104b90 0x7f294c19adc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f293c00e9c0 tx=0x7f293c00ee80 comp rx=0 tx=0).stop 2024-04-28T12:48:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.287+0000 7f2954200640 1 -- 172.21.15.84:0/2186076031 shutdown_connections 2024-04-28T12:48:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.287+0000 7f2954200640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f29300a3c30 0x7f29300a60f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.287+0000 7f2954200640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f294c106740 0x7f294c1a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.287+0000 7f2954200640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f294c105d80 0x7f294c19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.287+0000 7f2954200640 1 --2- 172.21.15.84:0/2186076031 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f294c104b90 0x7f294c19adc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.287+0000 7f2954200640 1 -- 172.21.15.84:0/2186076031 >> 172.21.15.84:0/2186076031 conn(0x7f294c100120 msgr2=0x7f294c109100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.287+0000 7f2954200640 1 -- 172.21.15.84:0/2186076031 shutdown_connections 2024-04-28T12:48:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:25.287+0000 7f2954200640 1 -- 172.21.15.84:0/2186076031 wait complete. 2024-04-28T12:48:25.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:25 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2186076031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:25.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:25 smithi084 ceph-mon[28081]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:25.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:25 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2186076031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:25.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:25 smithi161 ceph-mon[62387]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:26.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:25 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2186076031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:26.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:25 smithi179 ceph-mon[63029]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:26.712 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:26 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.84:6840/1196924806,v1:172.21.15.84:6841/1196924806] up:boot 2024-04-28T12:48:26.713 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:26 smithi161 ceph-mon[62387]: fsmap 11 up:standby 2024-04-28T12:48:26.713 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-28T12:48:26.713 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:26.713 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:26.713 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:26.713 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:26.713 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:26.713 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:26.713 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:26 smithi161 ceph-mon[62387]: Deploying daemon mds.b on smithi161 2024-04-28T12:48:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:26 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.84:6840/1196924806,v1:172.21.15.84:6841/1196924806] up:boot 2024-04-28T12:48:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:26 smithi179 ceph-mon[63029]: fsmap 11 up:standby 2024-04-28T12:48:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-28T12:48:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:26 smithi179 ceph-mon[63029]: Deploying daemon mds.b on smithi161 2024-04-28T12:48:27.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:26 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.84:6840/1196924806,v1:172.21.15.84:6841/1196924806] up:boot 2024-04-28T12:48:27.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:26 smithi084 ceph-mon[28081]: fsmap 11 up:standby 2024-04-28T12:48:27.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-28T12:48:27.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:27.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:27.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:27.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-28T12:48:27.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-28T12:48:27.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:27.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:26 smithi084 ceph-mon[28081]: Deploying daemon mds.b on smithi161 2024-04-28T12:48:27.301 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:27.519 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.523+0000 7ff155000640 1 -- 172.21.15.84:0/2091760391 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff150106540 msgr2=0x7ff15010cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:27.519 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.523+0000 7ff155000640 1 --2- 172.21.15.84:0/2091760391 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff150106540 0x7ff15010cdd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff148009ca0 tx=0x7ff14802a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:27.519 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.523+0000 7ff155000640 1 -- 172.21.15.84:0/2091760391 shutdown_connections 2024-04-28T12:48:27.519 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.523+0000 7ff155000640 1 --2- 172.21.15.84:0/2091760391 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff150106540 0x7ff15010cdd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:27.519 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.523+0000 7ff155000640 1 --2- 172.21.15.84:0/2091760391 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff150105b80 0x7ff150106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:27.519 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.523+0000 7ff155000640 1 --2- 172.21.15.84:0/2091760391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff150104990 0x7ff150104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:27.520 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.523+0000 7ff155000640 1 -- 172.21.15.84:0/2091760391 >> 172.21.15.84:0/2091760391 conn(0x7ff1500fff00 msgr2=0x7ff150102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:27.520 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.523+0000 7ff155000640 1 -- 172.21.15.84:0/2091760391 shutdown_connections 2024-04-28T12:48:27.520 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.523+0000 7ff155000640 1 -- 172.21.15.84:0/2091760391 wait complete. 2024-04-28T12:48:27.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.524+0000 7ff155000640 1 Processor -- start 2024-04-28T12:48:27.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff155000640 1 -- start start 2024-04-28T12:48:27.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff155000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff150104990 0x7ff15019ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:27.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff155000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff150105b80 0x7ff15019b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:27.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff155000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff150106540 0x7ff1501a2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:27.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff155000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff150110510 con 0x7ff150105b80 2024-04-28T12:48:27.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff155000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff150110390 con 0x7ff150106540 2024-04-28T12:48:27.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff155000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff150110690 con 0x7ff150104990 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff14f400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff150104990 0x7ff15019ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff14f400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff150104990 0x7ff15019ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:53024/0 (socket says 172.21.15.84:53024) 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff14f400640 1 -- 172.21.15.84:0/1948039168 learned_addr learned my addr 172.21.15.84:0/1948039168 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff146a00640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff150105b80 0x7ff15019b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff14fe00640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff150106540 0x7ff1501a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff14f400640 1 -- 172.21.15.84:0/1948039168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff150106540 msgr2=0x7ff1501a2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff14f400640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff150106540 0x7ff1501a2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff14f400640 1 -- 172.21.15.84:0/1948039168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff150105b80 msgr2=0x7ff15019b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff14f400640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff150105b80 0x7ff15019b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff14f400640 1 -- 172.21.15.84:0/1948039168 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1501a2770 con 0x7ff150104990 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.525+0000 7ff14fe00640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff150106540 0x7ff1501a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.526+0000 7ff14f400640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff150104990 0x7ff15019ab50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff14000e9c0 tx=0x7ff14000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.526+0000 7ff14cc00640 1 -- 172.21.15.84:0/1948039168 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14000cde0 con 0x7ff150104990 2024-04-28T12:48:27.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.526+0000 7ff155000640 1 -- 172.21.15.84:0/1948039168 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1501a2a60 con 0x7ff150104990 2024-04-28T12:48:27.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.526+0000 7ff14cc00640 1 -- 172.21.15.84:0/1948039168 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1400108e0 con 0x7ff150104990 2024-04-28T12:48:27.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.526+0000 7ff14cc00640 1 -- 172.21.15.84:0/1948039168 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1400188d0 con 0x7ff150104990 2024-04-28T12:48:27.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.526+0000 7ff155000640 1 -- 172.21.15.84:0/1948039168 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1501a2fb0 con 0x7ff150104990 2024-04-28T12:48:27.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.527+0000 7ff155000640 1 -- 172.21.15.84:0/1948039168 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff15010bcb0 con 0x7ff150104990 2024-04-28T12:48:27.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.529+0000 7ff14cc00640 1 -- 172.21.15.84:0/1948039168 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff14001f020 con 0x7ff150104990 2024-04-28T12:48:27.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.531+0000 7ff14cc00640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff1340a3bb0 0x7ff1340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:27.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.531+0000 7ff14cc00640 1 -- 172.21.15.84:0/1948039168 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff1400d0480 con 0x7ff150104990 2024-04-28T12:48:27.526 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.531+0000 7ff146a00640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff1340a3bb0 0x7ff1340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:27.527 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.532+0000 7ff146a00640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff1340a3bb0 0x7ff1340a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff15019c060 tx=0x7ff138005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:27.529 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.533+0000 7ff14cc00640 1 -- 172.21.15.84:0/1948039168 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff14009d260 con 0x7ff150104990 2024-04-28T12:48:27.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:27 smithi161 ceph-mon[62387]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:27.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.782+0000 7ff155000640 1 -- 172.21.15.84:0/1948039168 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff15002dd60 con 0x7ff150104990 2024-04-28T12:48:27.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.783+0000 7ff14cc00640 1 -- 172.21.15.84:0/1948039168 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9665 (secure 0 0 0) 0x7ff14009cb60 con 0x7ff150104990 2024-04-28T12:48:27.779 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:27.779 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6841/1196924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":1196924806},{"type":"v1","addr":"172.21.15.84:6841","nonce":1196924806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24562,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-28T12:48:27.779 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 12 2024-04-28T12:48:27.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.786+0000 7ff155000640 1 -- 172.21.15.84:0/1948039168 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff1340a3bb0 msgr2=0x7ff1340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:27.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.786+0000 7ff155000640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff1340a3bb0 0x7ff1340a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff15019c060 tx=0x7ff138005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:27.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.786+0000 7ff155000640 1 -- 172.21.15.84:0/1948039168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff150104990 msgr2=0x7ff15019ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:27.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.786+0000 7ff155000640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff150104990 0x7ff15019ab50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff14000e9c0 tx=0x7ff14000ee80 comp rx=0 tx=0).stop 2024-04-28T12:48:27.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.786+0000 7ff155000640 1 -- 172.21.15.84:0/1948039168 shutdown_connections 2024-04-28T12:48:27.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.786+0000 7ff155000640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff1340a3bb0 0x7ff1340a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:27.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.786+0000 7ff155000640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff150106540 0x7ff1501a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:27.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.786+0000 7ff155000640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff150105b80 0x7ff15019b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:27.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.786+0000 7ff155000640 1 --2- 172.21.15.84:0/1948039168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff150104990 0x7ff15019ab50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:27.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.786+0000 7ff155000640 1 -- 172.21.15.84:0/1948039168 >> 172.21.15.84:0/1948039168 conn(0x7ff1500fff00 msgr2=0x7ff150108e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:27.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.786+0000 7ff155000640 1 -- 172.21.15.84:0/1948039168 shutdown_connections 2024-04-28T12:48:27.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:27.787+0000 7ff155000640 1 -- 172.21.15.84:0/1948039168 wait complete. 2024-04-28T12:48:28.150 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 12:48:27 smithi161 systemd[1]: Starting Ceph mds.b for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T12:48:28.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:27 smithi179 ceph-mon[63029]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:28.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:27 smithi084 ceph-mon[28081]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:28.439 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 12:48:28 smithi161 podman[92446]: 2024-04-28 12:48:28.057968042 +0000 UTC m=+0.018077327 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T12:48:28.439 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 12:48:28 smithi161 podman[92446]: 2024-04-28 12:48:28.199436956 +0000 UTC m=+0.159546245 container create 15ab11a559f9636c7a77aa6c623cc2dc8ee17e988ea1cc7f66a385ed913847d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-28T12:48:28.439 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 12:48:28 smithi161 podman[92446]: 2024-04-28 12:48:28.443753713 +0000 UTC m=+0.403862999 container init 15ab11a559f9636c7a77aa6c623cc2dc8ee17e988ea1cc7f66a385ed913847d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-28T12:48:28.699 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 12:48:28 smithi161 podman[92446]: 2024-04-28 12:48:28.446387412 +0000 UTC m=+0.406496698 container start 15ab11a559f9636c7a77aa6c623cc2dc8ee17e988ea1cc7f66a385ed913847d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T12:48:28.699 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 12:48:28 smithi161 ceph-mds[92465]: -- 172.21.15.161:0/3689901508 <== mon.2 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5572e6c47a40 con 0x5572e6c1a800 2024-04-28T12:48:28.699 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 12:48:28 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b[92461]: starting mds.b at 2024-04-28T12:48:28.699 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 12:48:28 smithi161 bash[92446]: 15ab11a559f9636c7a77aa6c623cc2dc8ee17e988ea1cc7f66a385ed913847d7 2024-04-28T12:48:28.699 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 12:48:28 smithi161 systemd[1]: Started Ceph mds.b for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T12:48:28.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:28 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1948039168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:28.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:28.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:28.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:28.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:28.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:28 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:29.147 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:28 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1948039168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:29.147 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:29.147 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:29.148 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:29.148 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:29.148 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:28 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:29.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:28 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1948039168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:29.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:29.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:29.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:29.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:29.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:28 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:29.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:29.980 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:29 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] up:boot 2024-04-28T12:48:29.980 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:29 smithi161 ceph-mon[62387]: fsmap 12 up:standby 2024-04-28T12:48:29.980 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T12:48:29.980 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:29.981 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:29 smithi161 ceph-mon[62387]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:29.981 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:29.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:29.999+0000 7fe752e00640 1 -- 172.21.15.84:0/1153010238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe74c101050 msgr2=0x7fe74c101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:29.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:29.999+0000 7fe752e00640 1 --2- 172.21.15.84:0/1153010238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe74c101050 0x7fe74c101470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe73c009900 tx=0x7fe73c02a3f0 comp rx=0 tx=0).stop 2024-04-28T12:48:29.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:29.999+0000 7fe752e00640 1 -- 172.21.15.84:0/1153010238 shutdown_connections 2024-04-28T12:48:29.995 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:29.999+0000 7fe752e00640 1 --2- 172.21.15.84:0/1153010238 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe74c102350 0x7fe74c1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:29.995 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:29.999+0000 7fe752e00640 1 --2- 172.21.15.84:0/1153010238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe74c1019b0 0x7fe74c101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:29.995 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:29.999+0000 7fe752e00640 1 --2- 172.21.15.84:0/1153010238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe74c101050 0x7fe74c101470 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:29.995 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:29.999+0000 7fe752e00640 1 -- 172.21.15.84:0/1153010238 >> 172.21.15.84:0/1153010238 conn(0x7fe74c0fcbf0 msgr2=0x7fe74c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:29.995 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:29.999+0000 7fe752e00640 1 -- 172.21.15.84:0/1153010238 shutdown_connections 2024-04-28T12:48:29.995 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:29.999+0000 7fe752e00640 1 -- 172.21.15.84:0/1153010238 wait complete. 2024-04-28T12:48:29.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.000+0000 7fe752e00640 1 Processor -- start 2024-04-28T12:48:29.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.000+0000 7fe752e00640 1 -- start start 2024-04-28T12:48:29.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe752e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe74c101050 0x7fe74c19ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:29.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe752e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe74c1019b0 0x7fe74c19b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:29.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe752e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe74c102350 0x7fe74c1a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe752e00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c110490 con 0x7fe74c101050 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe752e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c110310 con 0x7fe74c102350 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe752e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c110610 con 0x7fe74c1019b0 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe751000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe74c1019b0 0x7fe74c19b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe751a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe74c101050 0x7fe74c19ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe751000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe74c1019b0 0x7fe74c19b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:42914/0 (socket says 172.21.15.84:42914) 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe751000640 1 -- 172.21.15.84:0/3961098050 learned_addr learned my addr 172.21.15.84:0/3961098050 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe752400640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe74c102350 0x7fe74c1a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe751000640 1 -- 172.21.15.84:0/3961098050 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe74c102350 msgr2=0x7fe74c1a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe751000640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe74c102350 0x7fe74c1a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe751000640 1 -- 172.21.15.84:0/3961098050 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe74c101050 msgr2=0x7fe74c19ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe751000640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe74c101050 0x7fe74c19ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:29.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.001+0000 7fe751000640 1 -- 172.21.15.84:0/3961098050 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74c1a2760 con 0x7fe74c1019b0 2024-04-28T12:48:29.999 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.002+0000 7fe751000640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe74c1019b0 0x7fe74c19b080 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe73800eb90 tx=0x7fe73800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:29.999 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.002+0000 7fe74a000640 1 -- 172.21.15.84:0/3961098050 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738004d20 con 0x7fe74c1019b0 2024-04-28T12:48:29.999 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.002+0000 7fe74a000640 1 -- 172.21.15.84:0/3961098050 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe73800ee90 con 0x7fe74c1019b0 2024-04-28T12:48:29.999 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.002+0000 7fe752e00640 1 -- 172.21.15.84:0/3961098050 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74c1a2a50 con 0x7fe74c1019b0 2024-04-28T12:48:29.999 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.002+0000 7fe74a000640 1 -- 172.21.15.84:0/3961098050 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738010670 con 0x7fe74c1019b0 2024-04-28T12:48:29.999 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.002+0000 7fe752e00640 1 -- 172.21.15.84:0/3961098050 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe74c1a2f50 con 0x7fe74c1019b0 2024-04-28T12:48:29.999 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.003+0000 7fe752e00640 1 -- 172.21.15.84:0/3961098050 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe74c10bf80 con 0x7fe74c1019b0 2024-04-28T12:48:30.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.005+0000 7fe74a000640 1 -- 172.21.15.84:0/3961098050 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe738005020 con 0x7fe74c1019b0 2024-04-28T12:48:30.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.006+0000 7fe74a000640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe7300a3bb0 0x7fe7300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:30.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.006+0000 7fe74a000640 1 -- 172.21.15.84:0/3961098050 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe7380d0620 con 0x7fe74c1019b0 2024-04-28T12:48:30.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.007+0000 7fe751a00640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe7300a3bb0 0x7fe7300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:30.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.007+0000 7fe751a00640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe7300a3bb0 0x7fe7300a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe73c000c00 tx=0x7fe73c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:30.005 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.010+0000 7fe74a000640 1 -- 172.21.15.84:0/3961098050 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe73809d400 con 0x7fe74c1019b0 2024-04-28T12:48:30.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:29 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] up:boot 2024-04-28T12:48:30.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:29 smithi084 ceph-mon[28081]: fsmap 12 up:standby 2024-04-28T12:48:30.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T12:48:30.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:30.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:29 smithi084 ceph-mon[28081]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:30.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:30.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:29 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] up:boot 2024-04-28T12:48:30.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:29 smithi179 ceph-mon[63029]: fsmap 12 up:standby 2024-04-28T12:48:30.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T12:48:30.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:30.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:29 smithi179 ceph-mon[63029]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:30.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:30.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.254+0000 7fe752e00640 1 -- 172.21.15.84:0/3961098050 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe74c063530 con 0x7fe74c1019b0 2024-04-28T12:48:30.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.255+0000 7fe74a000640 1 -- 172.21.15.84:0/3961098050 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10494 (secure 0 0 0) 0x7fe73809cd00 con 0x7fe74c1019b0 2024-04-28T12:48:30.251 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:30.251 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6841/1196924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":1196924806},{"type":"v1","addr":"172.21.15.84:6841","nonce":1196924806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24562,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24602,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/3812720642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":3812720642},{"type":"v1","addr":"172.21.15.161:6839","nonce":3812720642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-28T12:48:30.251 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 13 2024-04-28T12:48:30.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.258+0000 7fe752e00640 1 -- 172.21.15.84:0/3961098050 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe7300a3bb0 msgr2=0x7fe7300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:30.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.258+0000 7fe752e00640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe7300a3bb0 0x7fe7300a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe73c000c00 tx=0x7fe73c0314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.258+0000 7fe752e00640 1 -- 172.21.15.84:0/3961098050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe74c1019b0 msgr2=0x7fe74c19b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:30.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.258+0000 7fe752e00640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe74c1019b0 0x7fe74c19b080 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe73800eb90 tx=0x7fe73800c6a0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.258+0000 7fe752e00640 1 -- 172.21.15.84:0/3961098050 shutdown_connections 2024-04-28T12:48:30.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.258+0000 7fe752e00640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe7300a3bb0 0x7fe7300a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.258+0000 7fe752e00640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe74c102350 0x7fe74c1a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.258+0000 7fe752e00640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe74c1019b0 0x7fe74c19b080 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.258+0000 7fe752e00640 1 --2- 172.21.15.84:0/3961098050 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe74c101050 0x7fe74c19ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.258+0000 7fe752e00640 1 -- 172.21.15.84:0/3961098050 >> 172.21.15.84:0/3961098050 conn(0x7fe74c0fcbf0 msgr2=0x7fe74c0fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:30.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.259+0000 7fe752e00640 1 -- 172.21.15.84:0/3961098050 shutdown_connections 2024-04-28T12:48:30.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.259+0000 7fe752e00640 1 -- 172.21.15.84:0/3961098050 wait complete. 2024-04-28T12:48:30.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:48:30.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.464+0000 7f6a03200640 1 -- 172.21.15.84:0/1232204529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69fc108d90 msgr2=0x7f69fc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:30.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.464+0000 7f6a03200640 1 --2- 172.21.15.84:0/1232204529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69fc108d90 0x7f69fc1091b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009860 tx=0x7f69f002a340 comp rx=0 tx=0).stop 2024-04-28T12:48:30.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.464+0000 7f6a03200640 1 -- 172.21.15.84:0/1232204529 shutdown_connections 2024-04-28T12:48:30.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.464+0000 7f6a03200640 1 --2- 172.21.15.84:0/1232204529 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69fc10a940 0x7f69fc1111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.464+0000 7f6a03200640 1 --2- 172.21.15.84:0/1232204529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69fc109f80 0x7f69fc10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.464+0000 7f6a03200640 1 --2- 172.21.15.84:0/1232204529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69fc108d90 0x7f69fc1091b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.464+0000 7f6a03200640 1 -- 172.21.15.84:0/1232204529 >> 172.21.15.84:0/1232204529 conn(0x7f69fc077d30 msgr2=0x7f69fc07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:30.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.464+0000 7f6a03200640 1 -- 172.21.15.84:0/1232204529 shutdown_connections 2024-04-28T12:48:30.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.464+0000 7f6a03200640 1 -- 172.21.15.84:0/1232204529 wait complete. 2024-04-28T12:48:30.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.465+0000 7f6a03200640 1 Processor -- start 2024-04-28T12:48:30.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.465+0000 7f6a03200640 1 -- start start 2024-04-28T12:48:30.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a03200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69fc108d90 0x7f69fc19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:30.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a03200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69fc109f80 0x7f69fc19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:30.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a03200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69fc10a940 0x7f69fc1a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:30.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a03200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc114890 con 0x7f69fc109f80 2024-04-28T12:48:30.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a03200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc114710 con 0x7f69fc10a940 2024-04-28T12:48:30.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a03200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc114a10 con 0x7f69fc108d90 2024-04-28T12:48:30.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a01e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69fc108d90 0x7f69fc19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:30.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a01e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69fc108d90 0x7f69fc19efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:42930/0 (socket says 172.21.15.84:42930) 2024-04-28T12:48:30.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a01e00640 1 -- 172.21.15.84:0/788550746 learned_addr learned my addr 172.21.15.84:0/788550746 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a01400640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69fc109f80 0x7f69fc19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a02800640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69fc10a940 0x7f69fc1a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a01e00640 1 -- 172.21.15.84:0/788550746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69fc10a940 msgr2=0x7f69fc1a6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a01e00640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69fc10a940 0x7f69fc1a6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a01e00640 1 -- 172.21.15.84:0/788550746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69fc109f80 msgr2=0x7f69fc19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a01e00640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69fc109f80 0x7f69fc19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a01e00640 1 -- 172.21.15.84:0/788550746 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69fc1a6c00 con 0x7f69fc108d90 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.466+0000 7f6a01400640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69fc109f80 0x7f69fc19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.467+0000 7f6a01e00640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69fc108d90 0x7f69fc19efe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f69f00057c0 tx=0x7f69f0032da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.467+0000 7f69f6a00640 1 -- 172.21.15.84:0/788550746 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f0037040 con 0x7f69fc108d90 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.467+0000 7f69f6a00640 1 -- 172.21.15.84:0/788550746 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69f0030bd0 con 0x7f69fc108d90 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.467+0000 7f6a03200640 1 -- 172.21.15.84:0/788550746 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69fc1a6e90 con 0x7f69fc108d90 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.467+0000 7f69f6a00640 1 -- 172.21.15.84:0/788550746 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f0005280 con 0x7f69fc108d90 2024-04-28T12:48:30.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.467+0000 7f6a03200640 1 -- 172.21.15.84:0/788550746 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69fc1a7240 con 0x7f69fc108d90 2024-04-28T12:48:30.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.468+0000 7f6a03200640 1 -- 172.21.15.84:0/788550746 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69fc1101d0 con 0x7f69fc108d90 2024-04-28T12:48:30.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.470+0000 7f69f6a00640 1 -- 172.21.15.84:0/788550746 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f69f0043020 con 0x7f69fc108d90 2024-04-28T12:48:30.467 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.472+0000 7f69f6a00640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69e40a3bb0 0x7f69e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:30.467 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.472+0000 7f69f6a00640 1 -- 172.21.15.84:0/788550746 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f69f00ef520 con 0x7f69fc108d90 2024-04-28T12:48:30.467 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.472+0000 7f6a01400640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69e40a3bb0 0x7f69e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:30.468 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.473+0000 7f6a01400640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69e40a3bb0 0x7f69e40a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f69fc1a04f0 tx=0x7f69ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:30.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.476+0000 7f69f6a00640 1 -- 172.21.15.84:0/788550746 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f69f00bc300 con 0x7f69fc108d90 2024-04-28T12:48:30.700 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.705+0000 7f6a03200640 1 -- 172.21.15.84:0/788550746 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69fc063670 con 0x7f69fc108d90 2024-04-28T12:48:30.701 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.706+0000 7f69f6a00640 1 -- 172.21.15.84:0/788550746 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18445 (secure 0 0 0) 0x7f69f0030d80 con 0x7f69fc108d90 2024-04-28T12:48:30.701 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:30.702 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":81,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:47:36.668828+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:48:30.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.708+0000 7f6a03200640 1 -- 172.21.15.84:0/788550746 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69e40a3bb0 msgr2=0x7f69e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:30.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.708+0000 7f6a03200640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69e40a3bb0 0x7f69e40a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f69fc1a04f0 tx=0x7f69ec009260 comp rx=0 tx=0).stop 2024-04-28T12:48:30.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.708+0000 7f6a03200640 1 -- 172.21.15.84:0/788550746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69fc108d90 msgr2=0x7f69fc19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:30.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.708+0000 7f6a03200640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69fc108d90 0x7f69fc19efe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f69f00057c0 tx=0x7f69f0032da0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.708+0000 7f6a03200640 1 -- 172.21.15.84:0/788550746 shutdown_connections 2024-04-28T12:48:30.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.708+0000 7f6a03200640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69e40a3bb0 0x7f69e40a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.708+0000 7f6a03200640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69fc10a940 0x7f69fc1a6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.708+0000 7f6a03200640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69fc109f80 0x7f69fc19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.708+0000 7f6a03200640 1 --2- 172.21.15.84:0/788550746 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69fc108d90 0x7f69fc19efe0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.708+0000 7f6a03200640 1 -- 172.21.15.84:0/788550746 >> 172.21.15.84:0/788550746 conn(0x7f69fc077d30 msgr2=0x7f69fc10d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:30.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.708+0000 7f6a03200640 1 -- 172.21.15.84:0/788550746 shutdown_connections 2024-04-28T12:48:30.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.708+0000 7f6a03200640 1 -- 172.21.15.84:0/788550746 wait complete. 2024-04-28T12:48:30.724 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:43:15.608080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-28T12:48:30.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:48:30.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.920+0000 7f475bc00640 1 -- 172.21.15.84:0/1848795873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4754105d20 msgr2=0x7f4754108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:30.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.920+0000 7f475bc00640 1 --2- 172.21.15.84:0/1848795873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4754105d20 0x7f4754108110 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f473c009860 tx=0x7f473c02a340 comp rx=0 tx=0).stop 2024-04-28T12:48:30.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.920+0000 7f475bc00640 1 -- 172.21.15.84:0/1848795873 shutdown_connections 2024-04-28T12:48:30.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.920+0000 7f475bc00640 1 --2- 172.21.15.84:0/1848795873 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4754108650 0x7f475410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.920+0000 7f475bc00640 1 --2- 172.21.15.84:0/1848795873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4754105d20 0x7f4754108110 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.920+0000 7f475bc00640 1 --2- 172.21.15.84:0/1848795873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47541033f0 0x7f47541057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.920+0000 7f475bc00640 1 -- 172.21.15.84:0/1848795873 >> 172.21.15.84:0/1848795873 conn(0x7f47540fce20 msgr2=0x7f47540ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:30.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.920+0000 7f475bc00640 1 -- 172.21.15.84:0/1848795873 shutdown_connections 2024-04-28T12:48:30.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.920+0000 7f475bc00640 1 -- 172.21.15.84:0/1848795873 wait complete. 2024-04-28T12:48:30.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.921+0000 7f475bc00640 1 Processor -- start 2024-04-28T12:48:30.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.921+0000 7f475bc00640 1 -- start start 2024-04-28T12:48:30.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f475bc00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47541033f0 0x7f475419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:30.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f475bc00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4754105d20 0x7f475419b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f475bc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4754108650 0x7f47541a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f475bc00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f475410e4b0 con 0x7f4754105d20 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f475bc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f475410e330 con 0x7f4754108650 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f475bc00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f475410e630 con 0x7f47541033f0 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f4759200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47541033f0 0x7f475419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f474be00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4754105d20 0x7f475419b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f4759200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47541033f0 0x7f475419ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:42946/0 (socket says 172.21.15.84:42946) 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f4759200640 1 -- 172.21.15.84:0/2103147104 learned_addr learned my addr 172.21.15.84:0/2103147104 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f4759c00640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4754108650 0x7f47541a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f4759200640 1 -- 172.21.15.84:0/2103147104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4754108650 msgr2=0x7f47541a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f4759200640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4754108650 0x7f47541a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f4759200640 1 -- 172.21.15.84:0/2103147104 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4754105d20 msgr2=0x7f475419b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f4759200640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4754105d20 0x7f475419b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.922+0000 7f4759200640 1 -- 172.21.15.84:0/2103147104 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47541a2a20 con 0x7f47541033f0 2024-04-28T12:48:30.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.923+0000 7f4759200640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47541033f0 0x7f475419ae00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f474400e9f0 tx=0x7f474400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:30.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.923+0000 7f4752000640 1 -- 172.21.15.84:0/2103147104 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474400ccc0 con 0x7f47541033f0 2024-04-28T12:48:30.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.923+0000 7f4752000640 1 -- 172.21.15.84:0/2103147104 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47440054d0 con 0x7f47541033f0 2024-04-28T12:48:30.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.923+0000 7f475bc00640 1 -- 172.21.15.84:0/2103147104 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47541a2d10 con 0x7f47541033f0 2024-04-28T12:48:30.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.923+0000 7f4752000640 1 -- 172.21.15.84:0/2103147104 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744010430 con 0x7f47541033f0 2024-04-28T12:48:30.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.923+0000 7f475bc00640 1 -- 172.21.15.84:0/2103147104 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47541a3180 con 0x7f47541033f0 2024-04-28T12:48:30.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.924+0000 7f475bc00640 1 -- 172.21.15.84:0/2103147104 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f475410a1a0 con 0x7f47541033f0 2024-04-28T12:48:30.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.926+0000 7f4752000640 1 -- 172.21.15.84:0/2103147104 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f47440046c0 con 0x7f47541033f0 2024-04-28T12:48:30.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.927+0000 7f4752000640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f47380a3c30 0x7f47380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:30.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.928+0000 7f4752000640 1 -- 172.21.15.84:0/2103147104 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f47440d0200 con 0x7f47541033f0 2024-04-28T12:48:30.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.928+0000 7f474be00640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f47380a3c30 0x7f47380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:30.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.928+0000 7f474be00640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f47380a3c30 0x7f47380a60f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f473c009830 tx=0x7f473c030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:30.926 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:30.931+0000 7f4752000640 1 -- 172.21.15.84:0/2103147104 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f474409cfe0 con 0x7f47541033f0 2024-04-28T12:48:30.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:30 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3961098050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:30.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:30.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:30.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:30 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/788550746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:31.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.144+0000 7f475bc00640 1 -- 172.21.15.84:0/2103147104 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f47540635d0 con 0x7f47541033f0 2024-04-28T12:48:31.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.144+0000 7f4752000640 1 -- 172.21.15.84:0/2103147104 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f474409c8e0 con 0x7f47541033f0 2024-04-28T12:48:31.140 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-04-28T12:48:31.142 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.147+0000 7f475bc00640 1 -- 172.21.15.84:0/2103147104 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f47380a3c30 msgr2=0x7f47380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.142 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.147+0000 7f475bc00640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f47380a3c30 0x7f47380a60f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f473c009830 tx=0x7f473c030450 comp rx=0 tx=0).stop 2024-04-28T12:48:31.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.147+0000 7f475bc00640 1 -- 172.21.15.84:0/2103147104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47541033f0 msgr2=0x7f475419ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.147+0000 7f475bc00640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47541033f0 0x7f475419ae00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f474400e9f0 tx=0x7f474400eeb0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.147+0000 7f475bc00640 1 -- 172.21.15.84:0/2103147104 shutdown_connections 2024-04-28T12:48:31.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.147+0000 7f475bc00640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f47380a3c30 0x7f47380a60f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.147+0000 7f475bc00640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4754108650 0x7f47541a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.147+0000 7f475bc00640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4754105d20 0x7f475419b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.147+0000 7f475bc00640 1 --2- 172.21.15.84:0/2103147104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47541033f0 0x7f475419ae00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.147+0000 7f475bc00640 1 -- 172.21.15.84:0/2103147104 >> 172.21.15.84:0/2103147104 conn(0x7f47540fce20 msgr2=0x7f4754102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:31.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.147+0000 7f475bc00640 1 -- 172.21.15.84:0/2103147104 shutdown_connections 2024-04-28T12:48:31.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.147+0000 7f475bc00640 1 -- 172.21.15.84:0/2103147104 wait complete. 2024-04-28T12:48:31.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:30 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3961098050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:31.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:31.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:31.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:30 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/788550746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:31.165 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-28T12:48:31.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:30 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3961098050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:31.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:31.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:31.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:30 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/788550746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:31.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.366+0000 7fe6da800640 1 -- 172.21.15.84:0/1680268534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d41027d0 msgr2=0x7fe6d4102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.366+0000 7fe6da800640 1 --2- 172.21.15.84:0/1680268534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d41027d0 0x7fe6d4102bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009900 tx=0x7fe6c802a3f0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.366+0000 7fe6da800640 1 -- 172.21.15.84:0/1680268534 shutdown_connections 2024-04-28T12:48:31.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.366+0000 7fe6da800640 1 --2- 172.21.15.84:0/1680268534 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d4104380 0x7fe6d410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.366+0000 7fe6da800640 1 --2- 172.21.15.84:0/1680268534 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d41039c0 0x7fe6d4103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.366+0000 7fe6da800640 1 --2- 172.21.15.84:0/1680268534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d41027d0 0x7fe6d4102bf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.366+0000 7fe6da800640 1 -- 172.21.15.84:0/1680268534 >> 172.21.15.84:0/1680268534 conn(0x7fe6d40fdd40 msgr2=0x7fe6d4100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:31.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.366+0000 7fe6da800640 1 -- 172.21.15.84:0/1680268534 shutdown_connections 2024-04-28T12:48:31.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.366+0000 7fe6da800640 1 -- 172.21.15.84:0/1680268534 wait complete. 2024-04-28T12:48:31.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.367+0000 7fe6da800640 1 Processor -- start 2024-04-28T12:48:31.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.367+0000 7fe6da800640 1 -- start start 2024-04-28T12:48:31.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.368+0000 7fe6da800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d41027d0 0x7fe6d419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:31.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.368+0000 7fe6da800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d41039c0 0x7fe6d419b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:31.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.368+0000 7fe6da800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d4104380 0x7fe6d41a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:31.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.368+0000 7fe6da800640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d410e320 con 0x7fe6d4104380 2024-04-28T12:48:31.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.368+0000 7fe6da800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d410e1a0 con 0x7fe6d41039c0 2024-04-28T12:48:31.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.368+0000 7fe6da800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d410e4a0 con 0x7fe6d41027d0 2024-04-28T12:48:31.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.368+0000 7fe6d9400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d41027d0 0x7fe6d419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:31.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.368+0000 7fe6d9400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d41027d0 0x7fe6d419ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:42974/0 (socket says 172.21.15.84:42974) 2024-04-28T12:48:31.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.368+0000 7fe6d9400640 1 -- 172.21.15.84:0/1017078773 learned_addr learned my addr 172.21.15.84:0/1017078773 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:31.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.368+0000 7fe6d9e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d4104380 0x7fe6d41a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.368+0000 7fe6d8a00640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d41039c0 0x7fe6d419b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.369+0000 7fe6d9400640 1 -- 172.21.15.84:0/1017078773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d41039c0 msgr2=0x7fe6d419b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.369+0000 7fe6d9400640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d41039c0 0x7fe6d419b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.369+0000 7fe6d9400640 1 -- 172.21.15.84:0/1017078773 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d4104380 msgr2=0x7fe6d41a21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.369+0000 7fe6d9400640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d4104380 0x7fe6d41a21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.369+0000 7fe6d9400640 1 -- 172.21.15.84:0/1017078773 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d41a2860 con 0x7fe6d41027d0 2024-04-28T12:48:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.369+0000 7fe6d9400640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d41027d0 0x7fe6d419ac50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8031500 tx=0x7fe6c8003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.369+0000 7fe6d2000640 1 -- 172.21.15.84:0/1017078773 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c803f070 con 0x7fe6d41027d0 2024-04-28T12:48:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.369+0000 7fe6da800640 1 -- 172.21.15.84:0/1017078773 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d41a2af0 con 0x7fe6d41027d0 2024-04-28T12:48:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.369+0000 7fe6d2000640 1 -- 172.21.15.84:0/1017078773 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c8031c50 con 0x7fe6d41027d0 2024-04-28T12:48:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.369+0000 7fe6da800640 1 -- 172.21.15.84:0/1017078773 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d41a3030 con 0x7fe6d41027d0 2024-04-28T12:48:31.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.370+0000 7fe6d2000640 1 -- 172.21.15.84:0/1017078773 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8049060 con 0x7fe6d41027d0 2024-04-28T12:48:31.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.371+0000 7fe6da800640 1 -- 172.21.15.84:0/1017078773 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d41a3200 con 0x7fe6d41027d0 2024-04-28T12:48:31.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.373+0000 7fe6d2000640 1 -- 172.21.15.84:0/1017078773 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe6c8042070 con 0x7fe6d41027d0 2024-04-28T12:48:31.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.374+0000 7fe6d2000640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe6bc0a3c30 0x7fe6bc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:31.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.374+0000 7fe6d2000640 1 -- 172.21.15.84:0/1017078773 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe6c80f0940 con 0x7fe6d41027d0 2024-04-28T12:48:31.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.374+0000 7fe6d8a00640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe6bc0a3c30 0x7fe6bc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:31.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.375+0000 7fe6d8a00640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe6bc0a3c30 0x7fe6bc0a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0005d00 tx=0x7fe6c000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:31.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.376+0000 7fe6d2000640 1 -- 172.21.15.84:0/1017078773 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe6c80bd0d0 con 0x7fe6d41027d0 2024-04-28T12:48:31.589 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.594+0000 7fe6da800640 1 -- 172.21.15.84:0/1017078773 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe6d41a34f0 con 0x7fe6d41027d0 2024-04-28T12:48:31.590 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.594+0000 7fe6d2000640 1 -- 172.21.15.84:0/1017078773 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fe6c80c13d0 con 0x7fe6d41027d0 2024-04-28T12:48:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:31.590 INFO:teuthology.orchestra.run.smithi084.stdout:[ 2024-04-28T12:48:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-04-28T12:48:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: "poolnum": 1, 2024-04-28T12:48:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: "poolname": ".mgr" 2024-04-28T12:48:31.590 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-04-28T12:48:31.590 INFO:teuthology.orchestra.run.smithi084.stdout:] 2024-04-28T12:48:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.597+0000 7fe6da800640 1 -- 172.21.15.84:0/1017078773 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe6bc0a3c30 msgr2=0x7fe6bc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.597+0000 7fe6da800640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe6bc0a3c30 0x7fe6bc0a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0005d00 tx=0x7fe6c000b410 comp rx=0 tx=0).stop 2024-04-28T12:48:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.597+0000 7fe6da800640 1 -- 172.21.15.84:0/1017078773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d41027d0 msgr2=0x7fe6d419ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.597+0000 7fe6da800640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d41027d0 0x7fe6d419ac50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8031500 tx=0x7fe6c8003c10 comp rx=0 tx=0).stop 2024-04-28T12:48:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.597+0000 7fe6da800640 1 -- 172.21.15.84:0/1017078773 shutdown_connections 2024-04-28T12:48:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.597+0000 7fe6da800640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe6bc0a3c30 0x7fe6bc0a60f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.597+0000 7fe6da800640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d4104380 0x7fe6d41a21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.597+0000 7fe6da800640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d41039c0 0x7fe6d419b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.597+0000 7fe6da800640 1 --2- 172.21.15.84:0/1017078773 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d41027d0 0x7fe6d419ac50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.597+0000 7fe6da800640 1 -- 172.21.15.84:0/1017078773 >> 172.21.15.84:0/1017078773 conn(0x7fe6d40fdd40 msgr2=0x7fe6d4074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.597+0000 7fe6da800640 1 -- 172.21.15.84:0/1017078773 shutdown_connections 2024-04-28T12:48:31.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.598+0000 7fe6da800640 1 -- 172.21.15.84:0/1017078773 wait complete. 2024-04-28T12:48:31.613 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-28T12:48:31.613 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-28T12:48:31.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.803+0000 7f07dfe00640 1 -- 172.21.15.84:0/3697019462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07e0108e70 msgr2=0x7f07e0109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.803+0000 7f07dfe00640 1 --2- 172.21.15.84:0/3697019462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07e0108e70 0x7f07e0109290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009900 tx=0x7f07d002a3f0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.803+0000 7f07dfe00640 1 -- 172.21.15.84:0/3697019462 shutdown_connections 2024-04-28T12:48:31.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.803+0000 7f07dfe00640 1 --2- 172.21.15.84:0/3697019462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07e010a980 0x7f07e01112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.803+0000 7f07dfe00640 1 --2- 172.21.15.84:0/3697019462 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e0109fc0 0x7f07e010a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.803+0000 7f07dfe00640 1 --2- 172.21.15.84:0/3697019462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07e0108e70 0x7f07e0109290 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.803+0000 7f07dfe00640 1 -- 172.21.15.84:0/3697019462 >> 172.21.15.84:0/3697019462 conn(0x7f07e0077e00 msgr2=0x7f07e007a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:31.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.804+0000 7f07dfe00640 1 -- 172.21.15.84:0/3697019462 shutdown_connections 2024-04-28T12:48:31.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.804+0000 7f07dfe00640 1 -- 172.21.15.84:0/3697019462 wait complete. 2024-04-28T12:48:31.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.804+0000 7f07dfe00640 1 Processor -- start 2024-04-28T12:48:31.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.805+0000 7f07dfe00640 1 -- start start 2024-04-28T12:48:31.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.805+0000 7f07dfe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e0108e70 0x7f07e019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:31.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.805+0000 7f07dfe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07e0109fc0 0x7f07e019f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:31.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.805+0000 7f07dfe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07e010a980 0x7f07e01a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:31.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.805+0000 7f07dfe00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f07e01149f0 con 0x7f07e0108e70 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.805+0000 7f07dfe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f07e0114870 con 0x7f07e0109fc0 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.805+0000 7f07dfe00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f07e0114b70 con 0x7f07e010a980 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.805+0000 7f07de000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07e0109fc0 0x7f07e019f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.805+0000 7f07dea00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e0108e70 0x7f07e019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.805+0000 7f07de000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07e0109fc0 0x7f07e019f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58570/0 (socket says 172.21.15.84:58570) 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.805+0000 7f07de000640 1 -- 172.21.15.84:0/3127984197 learned_addr learned my addr 172.21.15.84:0/3127984197 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.805+0000 7f07df400640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07e010a980 0x7f07e01a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.806+0000 7f07df400640 1 -- 172.21.15.84:0/3127984197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07e0109fc0 msgr2=0x7f07e019f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.806+0000 7f07df400640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07e0109fc0 0x7f07e019f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.806+0000 7f07df400640 1 -- 172.21.15.84:0/3127984197 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e0108e70 msgr2=0x7f07e019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.806+0000 7f07df400640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e0108e70 0x7f07e019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.806+0000 7f07df400640 1 -- 172.21.15.84:0/3127984197 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e01a6cb0 con 0x7f07e010a980 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.806+0000 7f07de000640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07e0109fc0 0x7f07e019f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.806+0000 7f07df400640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07e010a980 0x7f07e01a6640 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f07d800ec50 tx=0x7f07d800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.806+0000 7f07d7400640 1 -- 172.21.15.84:0/3127984197 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8003b70 con 0x7f07e010a980 2024-04-28T12:48:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.806+0000 7f07dfe00640 1 -- 172.21.15.84:0/3127984197 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e01a6fa0 con 0x7f07e010a980 2024-04-28T12:48:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.806+0000 7f07d7400640 1 -- 172.21.15.84:0/3127984197 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07d8003d10 con 0x7f07e010a980 2024-04-28T12:48:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.806+0000 7f07d7400640 1 -- 172.21.15.84:0/3127984197 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8010830 con 0x7f07e010a980 2024-04-28T12:48:31.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.807+0000 7f07dfe00640 1 -- 172.21.15.84:0/3127984197 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e01a7410 con 0x7f07e010a980 2024-04-28T12:48:31.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.808+0000 7f07dfe00640 1 -- 172.21.15.84:0/3127984197 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07e0110180 con 0x7f07e010a980 2024-04-28T12:48:31.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.811+0000 7f07d7400640 1 -- 172.21.15.84:0/3127984197 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f07d8003ec0 con 0x7f07e010a980 2024-04-28T12:48:31.808 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.812+0000 7f07d7400640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f07c40a3bb0 0x7f07c40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:31.808 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.812+0000 7f07d7400640 1 -- 172.21.15.84:0/3127984197 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f07d80d04b0 con 0x7f07e010a980 2024-04-28T12:48:31.808 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.812+0000 7f07dea00640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f07c40a3bb0 0x7f07c40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:31.808 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.813+0000 7f07dea00640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f07c40a3bb0 0x7f07c40a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f07d0031500 tx=0x7f07d0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:31.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:31.816+0000 7f07d7400640 1 -- 172.21.15.84:0/3127984197 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f07d809d330 con 0x7f07e010a980 2024-04-28T12:48:31.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:31 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:31.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:31 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:31.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:31 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2103147104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:31 smithi084 ceph-mon[28081]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:31 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1017078773' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-28T12:48:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:31 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:31 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:31 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:31 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:31 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:32.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.026+0000 7f07dfe00640 1 -- 172.21.15.84:0/3127984197 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f07e002dd20 con 0x7f07e010a980 2024-04-28T12:48:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:31 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:31 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:31 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2103147104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:31 smithi179 ceph-mon[63029]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:31 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1017078773' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-28T12:48:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:31 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:31 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:31 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:31 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:31 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:32.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:31 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:32.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:31 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:32.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:31 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2103147104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:32.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:31 smithi161 ceph-mon[62387]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:32.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:31 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1017078773' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-28T12:48:32.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:31 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:32.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:31 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:32.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:31 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:32.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:31 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:32.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:31 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:32.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.815+0000 7f07d7400640 1 -- 172.21.15.84:0/3127984197 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f07d809cc30 con 0x7f07e010a980 2024-04-28T12:48:32.814 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'cephfs_metadata' created 2024-04-28T12:48:32.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.820+0000 7f07dfe00640 1 -- 172.21.15.84:0/3127984197 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f07c40a3bb0 msgr2=0x7f07c40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:32.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.820+0000 7f07dfe00640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f07c40a3bb0 0x7f07c40a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f07d0031500 tx=0x7f07d0005a20 comp rx=0 tx=0).stop 2024-04-28T12:48:32.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.820+0000 7f07dfe00640 1 -- 172.21.15.84:0/3127984197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07e010a980 msgr2=0x7f07e01a6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:32.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.820+0000 7f07dfe00640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07e010a980 0x7f07e01a6640 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f07d800ec50 tx=0x7f07d800c6a0 comp rx=0 tx=0).stop 2024-04-28T12:48:32.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.820+0000 7f07dfe00640 1 -- 172.21.15.84:0/3127984197 shutdown_connections 2024-04-28T12:48:32.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.820+0000 7f07dfe00640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f07c40a3bb0 0x7f07c40a6070 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f07d0031500 tx=0x7f07d0005a20 comp rx=0 tx=0).stop 2024-04-28T12:48:32.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.820+0000 7f07dfe00640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07e010a980 0x7f07e01a6640 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:32.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.820+0000 7f07dfe00640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07e0109fc0 0x7f07e019f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:32.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.820+0000 7f07dfe00640 1 --2- 172.21.15.84:0/3127984197 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e0108e70 0x7f07e019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:32.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.820+0000 7f07dfe00640 1 -- 172.21.15.84:0/3127984197 >> 172.21.15.84:0/3127984197 conn(0x7f07e0077e00 msgr2=0x7f07e010d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:32.818 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.822+0000 7f07dfe00640 1 -- 172.21.15.84:0/3127984197 shutdown_connections 2024-04-28T12:48:32.818 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:32.822+0000 7f07dfe00640 1 -- 172.21.15.84:0/3127984197 wait complete. 2024-04-28T12:48:32.836 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-28T12:48:32.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:32.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:32 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3127984197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T12:48:32.952 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:32 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T12:48:32.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:32.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:32 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3127984197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T12:48:32.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:32 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T12:48:33.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.017+0000 7f8c70600640 1 -- 172.21.15.84:0/3353304474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c680772f0 msgr2=0x7f8c68077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:33.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.017+0000 7f8c70600640 1 --2- 172.21.15.84:0/3353304474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c680772f0 0x7f8c68077710 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c009900 tx=0x7f8c5c02a410 comp rx=0 tx=0).stop 2024-04-28T12:48:33.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.017+0000 7f8c70600640 1 -- 172.21.15.84:0/3353304474 shutdown_connections 2024-04-28T12:48:33.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.017+0000 7f8c70600640 1 --2- 172.21.15.84:0/3353304474 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c68078ed0 0x7f8c68079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.017+0000 7f8c70600640 1 --2- 172.21.15.84:0/3353304474 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c68078530 0x7f8c68078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.017+0000 7f8c70600640 1 --2- 172.21.15.84:0/3353304474 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c680772f0 0x7f8c68077710 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.017+0000 7f8c70600640 1 -- 172.21.15.84:0/3353304474 >> 172.21.15.84:0/3353304474 conn(0x7f8c681000e0 msgr2=0x7f8c68102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:33.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.018+0000 7f8c70600640 1 -- 172.21.15.84:0/3353304474 shutdown_connections 2024-04-28T12:48:33.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.018+0000 7f8c70600640 1 -- 172.21.15.84:0/3353304474 wait complete. 2024-04-28T12:48:33.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.018+0000 7f8c70600640 1 Processor -- start 2024-04-28T12:48:33.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.019+0000 7f8c70600640 1 -- start start 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.019+0000 7f8c70600640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c680772f0 0x7f8c6819f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.019+0000 7f8c70600640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c68078530 0x7f8c6819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.019+0000 7f8c70600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c68078ed0 0x7f8c681a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.019+0000 7f8c70600640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8c68114b10 con 0x7f8c680772f0 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.019+0000 7f8c70600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8c68114990 con 0x7f8c68078ed0 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.019+0000 7f8c70600640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f8c68114c90 con 0x7f8c68078530 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.019+0000 7f8c6dc00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c680772f0 0x7f8c6819f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.019+0000 7f8c6d200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c68078530 0x7f8c6819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.019+0000 7f8c6dc00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c680772f0 0x7f8c6819f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50600/0 (socket says 172.21.15.84:50600) 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.019+0000 7f8c6dc00640 1 -- 172.21.15.84:0/2637110907 learned_addr learned my addr 172.21.15.84:0/2637110907 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.019+0000 7f8c6e600640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c68078ed0 0x7f8c681a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.020+0000 7f8c6d200640 1 -- 172.21.15.84:0/2637110907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c68078ed0 msgr2=0x7f8c681a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.020+0000 7f8c6d200640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c68078ed0 0x7f8c681a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.020+0000 7f8c6d200640 1 -- 172.21.15.84:0/2637110907 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c680772f0 msgr2=0x7f8c6819f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:33.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.020+0000 7f8c6d200640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c680772f0 0x7f8c6819f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.020+0000 7f8c6d200640 1 -- 172.21.15.84:0/2637110907 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c681a6de0 con 0x7f8c68078530 2024-04-28T12:48:33.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.020+0000 7f8c6d200640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c68078530 0x7f8c6819f700 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400e9f0 tx=0x7f8c5400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:33.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.020+0000 7f8c66000640 1 -- 172.21.15.84:0/2637110907 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5400ccc0 con 0x7f8c68078530 2024-04-28T12:48:33.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.020+0000 7f8c70600640 1 -- 172.21.15.84:0/2637110907 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c681a70d0 con 0x7f8c68078530 2024-04-28T12:48:33.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.020+0000 7f8c66000640 1 -- 172.21.15.84:0/2637110907 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c540108e0 con 0x7f8c68078530 2024-04-28T12:48:33.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.020+0000 7f8c66000640 1 -- 172.21.15.84:0/2637110907 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54018880 con 0x7f8c68078530 2024-04-28T12:48:33.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.020+0000 7f8c70600640 1 -- 172.21.15.84:0/2637110907 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c681a7620 con 0x7f8c68078530 2024-04-28T12:48:33.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.021+0000 7f8c70600640 1 -- 172.21.15.84:0/2637110907 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c68111700 con 0x7f8c68078530 2024-04-28T12:48:33.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.023+0000 7f8c66000640 1 -- 172.21.15.84:0/2637110907 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8c54010430 con 0x7f8c68078530 2024-04-28T12:48:33.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.025+0000 7f8c66000640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8c500a3bb0 0x7f8c500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:33.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.026+0000 7f8c66000640 1 -- 172.21.15.84:0/2637110907 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f8c540d0460 con 0x7f8c68078530 2024-04-28T12:48:33.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.026+0000 7f8c6dc00640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8c500a3bb0 0x7f8c500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:33.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.026+0000 7f8c6dc00640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8c500a3bb0 0x7f8c500a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0096c0 tx=0x7f8c5c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:33.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.028+0000 7f8c66000640 1 -- 172.21.15.84:0/2637110907 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8c5409ca80 con 0x7f8c68078530 2024-04-28T12:48:33.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:48:33.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:32 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3127984197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T12:48:33.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:32 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-28T12:48:33.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.229+0000 7f8c70600640 1 -- 172.21.15.84:0/2637110907 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f8c68063530 con 0x7f8c68078530 2024-04-28T12:48:33.820 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.824+0000 7f8c66000640 1 -- 172.21.15.84:0/2637110907 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7f8c540a0d80 con 0x7f8c68078530 2024-04-28T12:48:33.825 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'cephfs_data' created 2024-04-28T12:48:33.834 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.838+0000 7f8c70600640 1 -- 172.21.15.84:0/2637110907 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8c500a3bb0 msgr2=0x7f8c500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:33.834 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.838+0000 7f8c70600640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8c500a3bb0 0x7f8c500a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0096c0 tx=0x7f8c5c005a20 comp rx=0 tx=0).stop 2024-04-28T12:48:33.834 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.838+0000 7f8c70600640 1 -- 172.21.15.84:0/2637110907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c68078530 msgr2=0x7f8c6819f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:33.834 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.838+0000 7f8c70600640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c68078530 0x7f8c6819f700 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400e9f0 tx=0x7f8c5400eeb0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.837 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.841+0000 7f8c70600640 1 -- 172.21.15.84:0/2637110907 shutdown_connections 2024-04-28T12:48:33.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.841+0000 7f8c70600640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8c500a3bb0 0x7f8c500a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.841+0000 7f8c70600640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c68078ed0 0x7f8c681a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.841+0000 7f8c70600640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8c68078530 0x7f8c6819f700 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.841+0000 7f8c70600640 1 --2- 172.21.15.84:0/2637110907 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c680772f0 0x7f8c6819f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:33.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.841+0000 7f8c70600640 1 -- 172.21.15.84:0/2637110907 >> 172.21.15.84:0/2637110907 conn(0x7f8c681000e0 msgr2=0x7f8c68101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:33.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.842+0000 7f8c70600640 1 -- 172.21.15.84:0/2637110907 shutdown_connections 2024-04-28T12:48:33.838 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:33.842+0000 7f8c70600640 1 -- 172.21.15.84:0/2637110907 wait complete. 2024-04-28T12:48:33.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-28T12:48:34.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.048+0000 7f87aa000640 1 -- 172.21.15.84:0/2139222517 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87a4075390 msgr2=0x7f87a4075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:34.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.048+0000 7f87aa000640 1 --2- 172.21.15.84:0/2139222517 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87a4075390 0x7f87a4075810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8790009930 tx=0x7f879002a420 comp rx=0 tx=0).stop 2024-04-28T12:48:34.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.048+0000 7f87aa000640 1 -- 172.21.15.84:0/2139222517 shutdown_connections 2024-04-28T12:48:34.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.048+0000 7f87aa000640 1 --2- 172.21.15.84:0/2139222517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87a410ee90 0x7f87a4111280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:34.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.048+0000 7f87aa000640 1 --2- 172.21.15.84:0/2139222517 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87a4075390 0x7f87a4075810 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:34.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.048+0000 7f87aa000640 1 --2- 172.21.15.84:0/2139222517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87a4076970 0x7f87a4074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:34.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.048+0000 7f87aa000640 1 -- 172.21.15.84:0/2139222517 >> 172.21.15.84:0/2139222517 conn(0x7f87a40fde30 msgr2=0x7f87a4100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:34.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.048+0000 7f87aa000640 1 -- 172.21.15.84:0/2139222517 shutdown_connections 2024-04-28T12:48:34.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.048+0000 7f87aa000640 1 -- 172.21.15.84:0/2139222517 wait complete. 2024-04-28T12:48:34.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.050+0000 7f87aa000640 1 Processor -- start 2024-04-28T12:48:34.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.050+0000 7f87aa000640 1 -- start start 2024-04-28T12:48:34.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.050+0000 7f87aa000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87a4075390 0x7f87a419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:34.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.050+0000 7f87aa000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87a4076970 0x7f87a419f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:34.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.050+0000 7f87aa000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87a410ee90 0x7f87a41a6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:34.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.050+0000 7f87aa000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f87a4114970 con 0x7f87a4076970 2024-04-28T12:48:34.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.050+0000 7f87aa000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f87a41147f0 con 0x7f87a410ee90 2024-04-28T12:48:34.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.050+0000 7f87aa000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f87a4114af0 con 0x7f87a4075390 2024-04-28T12:48:34.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.051+0000 7f87a8c00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87a4075390 0x7f87a419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:34.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.051+0000 7f87a8c00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87a4075390 0x7f87a419f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:43036/0 (socket says 172.21.15.84:43036) 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.051+0000 7f87a3e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87a4076970 0x7f87a419f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.051+0000 7f87a8c00640 1 -- 172.21.15.84:0/1489160310 learned_addr learned my addr 172.21.15.84:0/1489160310 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.051+0000 7f87a9600640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87a410ee90 0x7f87a41a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.051+0000 7f87a8c00640 1 -- 172.21.15.84:0/1489160310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87a410ee90 msgr2=0x7f87a41a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.051+0000 7f87a8c00640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87a410ee90 0x7f87a41a6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.051+0000 7f87a8c00640 1 -- 172.21.15.84:0/1489160310 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87a4076970 msgr2=0x7f87a419f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.051+0000 7f87a8c00640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87a4076970 0x7f87a419f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.051+0000 7f87a8c00640 1 -- 172.21.15.84:0/1489160310 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a41a6ce0 con 0x7f87a4075390 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.051+0000 7f87a8c00640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87a4075390 0x7f87a419f0c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f879800ea60 tx=0x7f879800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.051+0000 7f87a1600640 1 -- 172.21.15.84:0/1489160310 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879800cde0 con 0x7f87a4075390 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.052+0000 7f87a1600640 1 -- 172.21.15.84:0/1489160310 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87980035f0 con 0x7f87a4075390 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.052+0000 7f87a1600640 1 -- 172.21.15.84:0/1489160310 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798010690 con 0x7f87a4075390 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.052+0000 7f87aa000640 1 -- 172.21.15.84:0/1489160310 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a41a6fd0 con 0x7f87a4075390 2024-04-28T12:48:34.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.052+0000 7f87aa000640 1 -- 172.21.15.84:0/1489160310 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a41a7430 con 0x7f87a4075390 2024-04-28T12:48:34.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.053+0000 7f87aa000640 1 -- 172.21.15.84:0/1489160310 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87a4105060 con 0x7f87a4075390 2024-04-28T12:48:34.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.055+0000 7f87a1600640 1 -- 172.21.15.84:0/1489160310 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f87980037a0 con 0x7f87a4075390 2024-04-28T12:48:34.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.057+0000 7f87a1600640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f878c0a3bb0 0x7f878c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:34.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.057+0000 7f87a1600640 1 -- 172.21.15.84:0/1489160310 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f87980d0580 con 0x7f87a4075390 2024-04-28T12:48:34.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.057+0000 7f87a3e00640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f878c0a3bb0 0x7f878c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:34.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.058+0000 7f87a3e00640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f878c0a3bb0 0x7f878c0a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8790006010 tx=0x7f87900314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:34.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.060+0000 7f87a1600640 1 -- 172.21.15.84:0/1489160310 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f879809d070 con 0x7f87a4075390 2024-04-28T12:48:34.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:33 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-28T12:48:34.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:33 smithi179 ceph-mon[63029]: osdmap e82: 12 total, 12 up, 12 in 2024-04-28T12:48:34.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:33 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2637110907' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-28T12:48:34.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:33 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-28T12:48:34.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:33 smithi179 ceph-mon[63029]: pgmap v314: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:34.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:33 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:34.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:33 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:34.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:33 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:34.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:33 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:34.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:33 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-28T12:48:34.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:33 smithi084 ceph-mon[28081]: osdmap e82: 12 total, 12 up, 12 in 2024-04-28T12:48:34.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:33 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2637110907' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-28T12:48:34.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:33 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-28T12:48:34.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:33 smithi084 ceph-mon[28081]: pgmap v314: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:34.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:34.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:34.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:34.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:34.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:33 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-28T12:48:34.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:33 smithi161 ceph-mon[62387]: osdmap e82: 12 total, 12 up, 12 in 2024-04-28T12:48:34.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:33 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2637110907' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-28T12:48:34.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:33 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-28T12:48:34.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:33 smithi161 ceph-mon[62387]: pgmap v314: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:34.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:33 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:34.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:33 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:34.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:33 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:34.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:33 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:34.328 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:34.333+0000 7f87aa000640 1 -- 172.21.15.84:0/1489160310 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f87a402dad0 con 0x7f87a4075390 2024-04-28T12:48:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:34 smithi179 ceph-mon[63029]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T12:48:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:34 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-28T12:48:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:34 smithi179 ceph-mon[63029]: osdmap e83: 12 total, 12 up, 12 in 2024-04-28T12:48:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:34 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1489160310' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-28T12:48:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:34 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-28T12:48:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:34 smithi179 ceph-mon[63029]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-28T12:48:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:34 smithi179 ceph-mon[63029]: osdmap e84: 12 total, 12 up, 12 in 2024-04-28T12:48:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:35.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:34 smithi084 ceph-mon[28081]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T12:48:35.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:34 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-28T12:48:35.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:34 smithi084 ceph-mon[28081]: osdmap e83: 12 total, 12 up, 12 in 2024-04-28T12:48:35.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:34 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1489160310' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-28T12:48:35.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:34 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-28T12:48:35.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:34 smithi084 ceph-mon[28081]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-28T12:48:35.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:35.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:34 smithi084 ceph-mon[28081]: osdmap e84: 12 total, 12 up, 12 in 2024-04-28T12:48:35.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:35.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:35.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:35.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:35.174 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.178+0000 7f87a1600640 1 -- 172.21.15.84:0/1489160310 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f879809c970 con 0x7f87a4075390 2024-04-28T12:48:35.174 INFO:teuthology.orchestra.run.smithi084.stderr:new fs with metadata pool 2 and data pool 3 2024-04-28T12:48:35.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.184+0000 7f87aa000640 1 -- 172.21.15.84:0/1489160310 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f878c0a3bb0 msgr2=0x7f878c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:35.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.184+0000 7f87aa000640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f878c0a3bb0 0x7f878c0a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8790006010 tx=0x7f87900314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:35.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.184+0000 7f87aa000640 1 -- 172.21.15.84:0/1489160310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87a4075390 msgr2=0x7f87a419f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:35.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.184+0000 7f87aa000640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87a4075390 0x7f87a419f0c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f879800ea60 tx=0x7f879800ef20 comp rx=0 tx=0).stop 2024-04-28T12:48:35.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.184+0000 7f87aa000640 1 -- 172.21.15.84:0/1489160310 shutdown_connections 2024-04-28T12:48:35.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.184+0000 7f87aa000640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f878c0a3bb0 0x7f878c0a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.184+0000 7f87aa000640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87a410ee90 0x7f87a41a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.184+0000 7f87aa000640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87a4076970 0x7f87a419f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.184+0000 7f87aa000640 1 --2- 172.21.15.84:0/1489160310 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87a4075390 0x7f87a419f0c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:35.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.184+0000 7f87aa000640 1 -- 172.21.15.84:0/1489160310 >> 172.21.15.84:0/1489160310 conn(0x7f87a40fde30 msgr2=0x7f87a4102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:35.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.185+0000 7f87aa000640 1 -- 172.21.15.84:0/1489160310 shutdown_connections 2024-04-28T12:48:35.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.185+0000 7f87aa000640 1 -- 172.21.15.84:0/1489160310 wait complete. 2024-04-28T12:48:35.202 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-28T12:48:35.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-28T12:48:35.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:34 smithi161 ceph-mon[62387]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T12:48:35.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:34 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-28T12:48:35.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:34 smithi161 ceph-mon[62387]: osdmap e83: 12 total, 12 up, 12 in 2024-04-28T12:48:35.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:34 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1489160310' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-28T12:48:35.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:34 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-28T12:48:35.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:34 smithi161 ceph-mon[62387]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-28T12:48:35.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:35.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:34 smithi161 ceph-mon[62387]: osdmap e84: 12 total, 12 up, 12 in 2024-04-28T12:48:35.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:35.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:48:35.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:48:35.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:48:35.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.409+0000 7f4549400640 1 -- 172.21.15.84:0/1961339740 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4544106740 msgr2=0x7f454410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:35.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.409+0000 7f4549400640 1 --2- 172.21.15.84:0/1961339740 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4544106740 0x7f454410cfd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f453c009ca0 tx=0x7f453c02a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:35.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.409+0000 7f4549400640 1 -- 172.21.15.84:0/1961339740 shutdown_connections 2024-04-28T12:48:35.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.409+0000 7f4549400640 1 --2- 172.21.15.84:0/1961339740 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4544106740 0x7f454410cfd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:35.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.409+0000 7f4549400640 1 --2- 172.21.15.84:0/1961339740 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4544105d80 0x7f4544106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:35.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.409+0000 7f4549400640 1 --2- 172.21.15.84:0/1961339740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4544104b90 0x7f4544104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:35.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.409+0000 7f4549400640 1 -- 172.21.15.84:0/1961339740 >> 172.21.15.84:0/1961339740 conn(0x7f4544100120 msgr2=0x7f4544102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:35.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.409+0000 7f4549400640 1 -- 172.21.15.84:0/1961339740 shutdown_connections 2024-04-28T12:48:35.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.409+0000 7f4549400640 1 -- 172.21.15.84:0/1961339740 wait complete. 2024-04-28T12:48:35.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.410+0000 7f4549400640 1 Processor -- start 2024-04-28T12:48:35.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.410+0000 7f4549400640 1 -- start start 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.410+0000 7f4549400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4544104b90 0x7f454419ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.410+0000 7f4549400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4544105d80 0x7f454419b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.410+0000 7f4549400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4544106740 0x7f45441a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.410+0000 7f4549400640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f45441106c0 con 0x7f4544105d80 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.410+0000 7f4549400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4544110540 con 0x7f4544104b90 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.410+0000 7f4549400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4544110840 con 0x7f4544106740 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f4541e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4544105d80 0x7f454419b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f4542800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4544104b90 0x7f454419ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f4541e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4544105d80 0x7f454419b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50634/0 (socket says 172.21.15.84:50634) 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f4541e00640 1 -- 172.21.15.84:0/4101278739 learned_addr learned my addr 172.21.15.84:0/4101278739 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f4543200640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4544106740 0x7f45441a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f4542800640 1 -- 172.21.15.84:0/4101278739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4544106740 msgr2=0x7f45441a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f4542800640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4544106740 0x7f45441a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f4542800640 1 -- 172.21.15.84:0/4101278739 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4544105d80 msgr2=0x7f454419b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:35.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f4542800640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4544105d80 0x7f454419b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:35.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f4542800640 1 -- 172.21.15.84:0/4101278739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45441a2960 con 0x7f4544104b90 2024-04-28T12:48:35.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f4541e00640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4544105d80 0x7f454419b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:35.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f4542800640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4544104b90 0x7f454419ad40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f453400e9c0 tx=0x7f453400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:35.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.411+0000 7f453b400640 1 -- 172.21.15.84:0/4101278739 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453400ccc0 con 0x7f4544104b90 2024-04-28T12:48:35.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.412+0000 7f453b400640 1 -- 172.21.15.84:0/4101278739 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45340035f0 con 0x7f4544104b90 2024-04-28T12:48:35.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.412+0000 7f453b400640 1 -- 172.21.15.84:0/4101278739 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453400ce60 con 0x7f4544104b90 2024-04-28T12:48:35.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.412+0000 7f4549400640 1 -- 172.21.15.84:0/4101278739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45441a2c50 con 0x7f4544104b90 2024-04-28T12:48:35.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.412+0000 7f4549400640 1 -- 172.21.15.84:0/4101278739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45441a31a0 con 0x7f4544104b90 2024-04-28T12:48:35.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.413+0000 7f4549400640 1 -- 172.21.15.84:0/4101278739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f454410bf80 con 0x7f4544104b90 2024-04-28T12:48:35.410 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.415+0000 7f453b400640 1 -- 172.21.15.84:0/4101278739 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4534004000 con 0x7f4544104b90 2024-04-28T12:48:35.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.416+0000 7f453b400640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f45280a3c30 0x7f45280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:35.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.416+0000 7f453b400640 1 -- 172.21.15.84:0/4101278739 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f45340d06b0 con 0x7f4544104b90 2024-04-28T12:48:35.412 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.416+0000 7f4541e00640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f45280a3c30 0x7f45280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:35.412 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.417+0000 7f4541e00640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f45280a3c30 0x7f45280a60f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f454419c250 tx=0x7f452c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:35.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.420+0000 7f453b400640 1 -- 172.21.15.84:0/4101278739 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f453409d140 con 0x7f4544104b90 2024-04-28T12:48:35.630 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:35.635+0000 7f4549400640 1 -- 172.21.15.84:0/4101278739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f454402dad0 con 0x7f4544104b90 2024-04-28T12:48:36.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:36 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-28T12:48:36.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:36 smithi161 ceph-mon[62387]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-28T12:48:36.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:36 smithi161 ceph-mon[62387]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-28T12:48:36.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:36 smithi161 ceph-mon[62387]: osdmap e85: 12 total, 12 up, 12 in 2024-04-28T12:48:36.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:36 smithi161 ceph-mon[62387]: pgmap v318: 97 pgs: 2 creating+peering, 47 active+clean, 48 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-28T12:48:36.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:36 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4101278739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-28T12:48:36.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:36 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-28T12:48:36.537 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.542+0000 7f453b400640 1 -- 172.21.15.84:0/4101278739 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) ==== 157+0+0 (secure 0 0 0) 0x7f453409ca40 con 0x7f4544104b90 2024-04-28T12:48:36.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.546+0000 7f4549400640 1 -- 172.21.15.84:0/4101278739 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f45280a3c30 msgr2=0x7f45280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:36.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.546+0000 7f4549400640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f45280a3c30 0x7f45280a60f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f454419c250 tx=0x7f452c005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:36.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.546+0000 7f4549400640 1 -- 172.21.15.84:0/4101278739 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4544104b90 msgr2=0x7f454419ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:36.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.546+0000 7f4549400640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4544104b90 0x7f454419ad40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f453400e9c0 tx=0x7f453400ee80 comp rx=0 tx=0).stop 2024-04-28T12:48:36.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.546+0000 7f4549400640 1 -- 172.21.15.84:0/4101278739 shutdown_connections 2024-04-28T12:48:36.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.546+0000 7f4549400640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f45280a3c30 0x7f45280a60f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.546+0000 7f4549400640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4544106740 0x7f45441a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.546+0000 7f4549400640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4544105d80 0x7f454419b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.546+0000 7f4549400640 1 --2- 172.21.15.84:0/4101278739 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4544104b90 0x7f454419ad40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.546+0000 7f4549400640 1 -- 172.21.15.84:0/4101278739 >> 172.21.15.84:0/4101278739 conn(0x7f4544100120 msgr2=0x7f4544109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:36.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.546+0000 7f4549400640 1 -- 172.21.15.84:0/4101278739 shutdown_connections 2024-04-28T12:48:36.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.546+0000 7f4549400640 1 -- 172.21.15.84:0/4101278739 wait complete. 2024-04-28T12:48:36.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:36 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-28T12:48:36.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:36 smithi084 ceph-mon[28081]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-28T12:48:36.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:36 smithi084 ceph-mon[28081]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-28T12:48:36.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:36 smithi084 ceph-mon[28081]: osdmap e85: 12 total, 12 up, 12 in 2024-04-28T12:48:36.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:36 smithi084 ceph-mon[28081]: pgmap v318: 97 pgs: 2 creating+peering, 47 active+clean, 48 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-28T12:48:36.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:36 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4101278739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-28T12:48:36.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:36 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-28T12:48:36.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-28T12:48:36.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:36 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-28T12:48:36.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:36 smithi179 ceph-mon[63029]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-28T12:48:36.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:36 smithi179 ceph-mon[63029]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-28T12:48:36.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:36 smithi179 ceph-mon[63029]: osdmap e85: 12 total, 12 up, 12 in 2024-04-28T12:48:36.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:36 smithi179 ceph-mon[63029]: pgmap v318: 97 pgs: 2 creating+peering, 47 active+clean, 48 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-28T12:48:36.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:36 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4101278739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-28T12:48:36.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:36 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-28T12:48:36.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.774+0000 7fbc7e400640 1 -- 172.21.15.84:0/1313956752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc78105ba0 msgr2=0x7fbc78106020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:36.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.774+0000 7fbc7e400640 1 --2- 172.21.15.84:0/1313956752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc78105ba0 0x7fbc78106020 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009860 tx=0x7fbc6802a340 comp rx=0 tx=0).stop 2024-04-28T12:48:36.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.775+0000 7fbc7e400640 1 -- 172.21.15.84:0/1313956752 shutdown_connections 2024-04-28T12:48:36.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.775+0000 7fbc7e400640 1 --2- 172.21.15.84:0/1313956752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc78106560 0x7fbc7810cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.775+0000 7fbc7e400640 1 --2- 172.21.15.84:0/1313956752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc78105ba0 0x7fbc78106020 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.775+0000 7fbc7e400640 1 --2- 172.21.15.84:0/1313956752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc781049b0 0x7fbc78104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.775+0000 7fbc7e400640 1 -- 172.21.15.84:0/1313956752 >> 172.21.15.84:0/1313956752 conn(0x7fbc780fff40 msgr2=0x7fbc78102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:36.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.775+0000 7fbc7e400640 1 -- 172.21.15.84:0/1313956752 shutdown_connections 2024-04-28T12:48:36.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.775+0000 7fbc7e400640 1 -- 172.21.15.84:0/1313956752 wait complete. 2024-04-28T12:48:36.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.776+0000 7fbc7e400640 1 Processor -- start 2024-04-28T12:48:36.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.776+0000 7fbc7e400640 1 -- start start 2024-04-28T12:48:36.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.776+0000 7fbc7e400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc781049b0 0x7fbc7819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:36.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.776+0000 7fbc7e400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc78105ba0 0x7fbc7819b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:36.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.776+0000 7fbc7e400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc78106560 0x7fbc781a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:36.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.776+0000 7fbc7e400640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbc781104e0 con 0x7fbc78105ba0 2024-04-28T12:48:36.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.776+0000 7fbc7e400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbc78110360 con 0x7fbc781049b0 2024-04-28T12:48:36.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.776+0000 7fbc7e400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbc78110660 con 0x7fbc78106560 2024-04-28T12:48:36.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.777+0000 7fbc77e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc78105ba0 0x7fbc7819b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:36.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.777+0000 7fbc7d000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc781049b0 0x7fbc7819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:36.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.777+0000 7fbc7da00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc78106560 0x7fbc781a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:36.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.777+0000 7fbc7d000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc781049b0 0x7fbc7819abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58648/0 (socket says 172.21.15.84:58648) 2024-04-28T12:48:36.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.777+0000 7fbc7d000640 1 -- 172.21.15.84:0/835899750 learned_addr learned my addr 172.21.15.84:0/835899750 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:36.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.777+0000 7fbc7d000640 1 -- 172.21.15.84:0/835899750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc78106560 msgr2=0x7fbc781a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:36.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.777+0000 7fbc7d000640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc78106560 0x7fbc781a2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.777+0000 7fbc7d000640 1 -- 172.21.15.84:0/835899750 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc78105ba0 msgr2=0x7fbc7819b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:36.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.777+0000 7fbc7d000640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc78105ba0 0x7fbc7819b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:36.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.777+0000 7fbc7d000640 1 -- 172.21.15.84:0/835899750 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc781a27d0 con 0x7fbc781049b0 2024-04-28T12:48:36.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.777+0000 7fbc7da00640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc78106560 0x7fbc781a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:36.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.777+0000 7fbc7d000640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc781049b0 0x7fbc7819abb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00e9a0 tx=0x7fbc6c00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:36.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.778+0000 7fbc75600640 1 -- 172.21.15.84:0/835899750 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00cd80 con 0x7fbc781049b0 2024-04-28T12:48:36.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.778+0000 7fbc7e400640 1 -- 172.21.15.84:0/835899750 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc781a2ac0 con 0x7fbc781049b0 2024-04-28T12:48:36.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.778+0000 7fbc75600640 1 -- 172.21.15.84:0/835899750 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c0108e0 con 0x7fbc781049b0 2024-04-28T12:48:36.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.778+0000 7fbc75600640 1 -- 172.21.15.84:0/835899750 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c018960 con 0x7fbc781049b0 2024-04-28T12:48:36.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.778+0000 7fbc7e400640 1 -- 172.21.15.84:0/835899750 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc781a3030 con 0x7fbc781049b0 2024-04-28T12:48:36.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.779+0000 7fbc7e400640 1 -- 172.21.15.84:0/835899750 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7810bd50 con 0x7fbc781049b0 2024-04-28T12:48:36.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.781+0000 7fbc75600640 1 -- 172.21.15.84:0/835899750 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbc6c010a90 con 0x7fbc781049b0 2024-04-28T12:48:36.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.782+0000 7fbc75600640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbc600a3bb0 0x7fbc600a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:36.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.782+0000 7fbc75600640 1 -- 172.21.15.84:0/835899750 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8277+0+0 (secure 0 0 0) 0x7fbc6c0d0970 con 0x7fbc781049b0 2024-04-28T12:48:36.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.783+0000 7fbc77e00640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbc600a3bb0 0x7fbc600a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:36.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.783+0000 7fbc77e00640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbc600a3bb0 0x7fbc600a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbc68000c00 tx=0x7fbc68030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:36.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.785+0000 7fbc75600640 1 -- 172.21.15.84:0/835899750 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbc6c09d2c0 con 0x7fbc781049b0 2024-04-28T12:48:36.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:36.985+0000 7fbc7e400640 1 -- 172.21.15.84:0/835899750 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fbc7802dad0 con 0x7fbc781049b0 2024-04-28T12:48:37.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:37 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] up:active 2024-04-28T12:48:37.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:37 smithi161 ceph-mon[62387]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:48:37.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:37 smithi161 ceph-mon[62387]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:48:37.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:37 smithi161 ceph-mon[62387]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-28T12:48:37.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:37 smithi161 ceph-mon[62387]: Cluster is now healthy 2024-04-28T12:48:37.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:37 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-28T12:48:37.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:37 smithi161 ceph-mon[62387]: osdmap e86: 12 total, 12 up, 12 in 2024-04-28T12:48:37.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:37 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/835899750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T12:48:37.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:37 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T12:48:37.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:37 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] up:active 2024-04-28T12:48:37.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:37 smithi179 ceph-mon[63029]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:48:37.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:37 smithi179 ceph-mon[63029]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:48:37.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:37 smithi179 ceph-mon[63029]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-28T12:48:37.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:37 smithi179 ceph-mon[63029]: Cluster is now healthy 2024-04-28T12:48:37.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:37 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-28T12:48:37.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:37 smithi179 ceph-mon[63029]: osdmap e86: 12 total, 12 up, 12 in 2024-04-28T12:48:37.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:37 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/835899750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T12:48:37.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:37 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T12:48:37.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:37 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] up:active 2024-04-28T12:48:37.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:37 smithi084 ceph-mon[28081]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:48:37.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:37 smithi084 ceph-mon[28081]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:48:37.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:37 smithi084 ceph-mon[28081]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-28T12:48:37.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:37 smithi084 ceph-mon[28081]: Cluster is now healthy 2024-04-28T12:48:37.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:37 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-28T12:48:37.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:37 smithi084 ceph-mon[28081]: osdmap e86: 12 total, 12 up, 12 in 2024-04-28T12:48:37.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:37 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/835899750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T12:48:37.668 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:37 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-28T12:48:38.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.562+0000 7fbc75600640 1 -- 172.21.15.84:0/835899750 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) ==== 228+0+0 (secure 0 0 0) 0x7fbc6c09cbc0 con 0x7fbc781049b0 2024-04-28T12:48:38.558 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'cephfs_data_ec' created 2024-04-28T12:48:38.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.565+0000 7fbc7e400640 1 -- 172.21.15.84:0/835899750 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbc600a3bb0 msgr2=0x7fbc600a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:38.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.565+0000 7fbc7e400640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbc600a3bb0 0x7fbc600a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbc68000c00 tx=0x7fbc68030470 comp rx=0 tx=0).stop 2024-04-28T12:48:38.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.566+0000 7fbc7e400640 1 -- 172.21.15.84:0/835899750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc781049b0 msgr2=0x7fbc7819abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:38.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.566+0000 7fbc7e400640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc781049b0 0x7fbc7819abb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00e9a0 tx=0x7fbc6c00ee60 comp rx=0 tx=0).stop 2024-04-28T12:48:38.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.566+0000 7fbc7e400640 1 -- 172.21.15.84:0/835899750 shutdown_connections 2024-04-28T12:48:38.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.566+0000 7fbc7e400640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbc600a3bb0 0x7fbc600a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.566+0000 7fbc7e400640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc78106560 0x7fbc781a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.566+0000 7fbc7e400640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc78105ba0 0x7fbc7819b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.566+0000 7fbc7e400640 1 --2- 172.21.15.84:0/835899750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc781049b0 0x7fbc7819abb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.566+0000 7fbc7e400640 1 -- 172.21.15.84:0/835899750 >> 172.21.15.84:0/835899750 conn(0x7fbc780fff40 msgr2=0x7fbc78108f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:38.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.566+0000 7fbc7e400640 1 -- 172.21.15.84:0/835899750 shutdown_connections 2024-04-28T12:48:38.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.566+0000 7fbc7e400640 1 -- 172.21.15.84:0/835899750 wait complete. 2024-04-28T12:48:38.589 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-28T12:48:38.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.775+0000 7f12f7e00640 1 -- 172.21.15.84:0/555192152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f810a980 msgr2=0x7f12f81112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:38.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.775+0000 7f12f7e00640 1 --2- 172.21.15.84:0/555192152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f810a980 0x7f12f81112a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f12f0009ca0 tx=0x7f12f002a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.775+0000 7f12f7e00640 1 -- 172.21.15.84:0/555192152 shutdown_connections 2024-04-28T12:48:38.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.775+0000 7f12f7e00640 1 --2- 172.21.15.84:0/555192152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f810a980 0x7f12f81112a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.775+0000 7f12f7e00640 1 --2- 172.21.15.84:0/555192152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12f8109fc0 0x7f12f810a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.775+0000 7f12f7e00640 1 --2- 172.21.15.84:0/555192152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12f8108e70 0x7f12f8109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.775+0000 7f12f7e00640 1 -- 172.21.15.84:0/555192152 >> 172.21.15.84:0/555192152 conn(0x7f12f8077e00 msgr2=0x7f12f807a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:38.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.775+0000 7f12f7e00640 1 -- 172.21.15.84:0/555192152 shutdown_connections 2024-04-28T12:48:38.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.775+0000 7f12f7e00640 1 -- 172.21.15.84:0/555192152 wait complete. 2024-04-28T12:48:38.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.776+0000 7f12f7e00640 1 Processor -- start 2024-04-28T12:48:38.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.776+0000 7f12f7e00640 1 -- start start 2024-04-28T12:48:38.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.776+0000 7f12f7e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12f8108e70 0x7f12f819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:38.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.776+0000 7f12f7e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12f8109fc0 0x7f12f819f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:38.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.776+0000 7f12f7e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f810a980 0x7f12f81a6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:38.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.776+0000 7f12f7e00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f12f8114a20 con 0x7f12f8109fc0 2024-04-28T12:48:38.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.776+0000 7f12f7e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f12f81148a0 con 0x7f12f810a980 2024-04-28T12:48:38.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.776+0000 7f12f7e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f12f8114ba0 con 0x7f12f8108e70 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.777+0000 7f12efe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12f8109fc0 0x7f12f819f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.777+0000 7f12f7400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f810a980 0x7f12f81a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.777+0000 7f12efe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12f8109fc0 0x7f12f819f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50674/0 (socket says 172.21.15.84:50674) 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.777+0000 7f12f7400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f810a980 0x7f12f81a6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58664/0 (socket says 172.21.15.84:58664) 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.777+0000 7f12f6a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12f8108e70 0x7f12f819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.777+0000 7f12efe00640 1 -- 172.21.15.84:0/293149857 learned_addr learned my addr 172.21.15.84:0/293149857 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.777+0000 7f12f6a00640 1 -- 172.21.15.84:0/293149857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f810a980 msgr2=0x7f12f81a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.777+0000 7f12f6a00640 1 --2- 172.21.15.84:0/293149857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f810a980 0x7f12f81a6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.777+0000 7f12f6a00640 1 -- 172.21.15.84:0/293149857 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12f8109fc0 msgr2=0x7f12f819f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.777+0000 7f12f6a00640 1 --2- 172.21.15.84:0/293149857 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12f8109fc0 0x7f12f819f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.777+0000 7f12f6a00640 1 -- 172.21.15.84:0/293149857 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f81a6bf0 con 0x7f12f8108e70 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.777+0000 7f12f6a00640 1 --2- 172.21.15.84:0/293149857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12f8108e70 0x7f12f819efd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f12e8005fd0 tx=0x7f12e800dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:38.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.778+0000 7f12ef400640 1 -- 172.21.15.84:0/293149857 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8009c30 con 0x7f12f8108e70 2024-04-28T12:48:38.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.778+0000 7f12ef400640 1 -- 172.21.15.84:0/293149857 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12e8009dd0 con 0x7f12f8108e70 2024-04-28T12:48:38.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.778+0000 7f12f7e00640 1 -- 172.21.15.84:0/293149857 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f81a6ee0 con 0x7f12f8108e70 2024-04-28T12:48:38.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.778+0000 7f12ef400640 1 -- 172.21.15.84:0/293149857 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8010460 con 0x7f12f8108e70 2024-04-28T12:48:38.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.778+0000 7f12f7e00640 1 -- 172.21.15.84:0/293149857 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f81a7340 con 0x7f12f8108e70 2024-04-28T12:48:38.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.779+0000 7f12f7e00640 1 -- 172.21.15.84:0/293149857 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f8069ac0 con 0x7f12f8108e70 2024-04-28T12:48:38.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.781+0000 7f12ef400640 1 -- 172.21.15.84:0/293149857 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f12e8003e80 con 0x7f12f8108e70 2024-04-28T12:48:38.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.782+0000 7f12ef400640 1 --2- 172.21.15.84:0/293149857 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f12dc0a3bb0 0x7f12dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:38.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.783+0000 7f12ef400640 1 -- 172.21.15.84:0/293149857 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f12e80d05d0 con 0x7f12f8108e70 2024-04-28T12:48:38.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.783+0000 7f12efe00640 1 --2- 172.21.15.84:0/293149857 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f12dc0a3bb0 0x7f12dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:38.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.783+0000 7f12efe00640 1 --2- 172.21.15.84:0/293149857 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f12dc0a3bb0 0x7f12dc0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f12f81a04e0 tx=0x7f12e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:38.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.787+0000 7f12ef400640 1 -- 172.21.15.84:0/293149857 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f12e8017020 con 0x7f12f8108e70 2024-04-28T12:48:38.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:38 smithi179 ceph-mon[63029]: osdmap e87: 12 total, 12 up, 12 in 2024-04-28T12:48:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:38 smithi179 ceph-mon[63029]: pgmap v321: 97 pgs: 2 creating+peering, 82 active+clean, 13 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 271 B/s wr, 6 op/s 2024-04-28T12:48:38.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:38 smithi084 ceph-mon[28081]: osdmap e87: 12 total, 12 up, 12 in 2024-04-28T12:48:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:38 smithi084 ceph-mon[28081]: pgmap v321: 97 pgs: 2 creating+peering, 82 active+clean, 13 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 271 B/s wr, 6 op/s 2024-04-28T12:48:38.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:38.985+0000 7f12f7e00640 1 -- 172.21.15.84:0/293149857 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f12f8063610 con 0x7f12f8108e70 2024-04-28T12:48:38.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:38 smithi161 ceph-mon[62387]: osdmap e87: 12 total, 12 up, 12 in 2024-04-28T12:48:38.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:38 smithi161 ceph-mon[62387]: pgmap v321: 97 pgs: 2 creating+peering, 82 active+clean, 13 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 271 B/s wr, 6 op/s 2024-04-28T12:48:38.992 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 12:48:38 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b[92461]: 2024-04-28T12:48:38.714+0000 7f630a6ec700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-28T12:48:39.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.578+0000 7f12ef400640 1 -- 172.21.15.84:0/293149857 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) ==== 169+0+0 (secure 0 0 0) 0x7f12e809c6e0 con 0x7f12f8108e70 2024-04-28T12:48:39.579 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 4 allow_ec_overwrites to true 2024-04-28T12:48:39.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.586+0000 7f12f7e00640 1 -- 172.21.15.84:0/293149857 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f12dc0a3bb0 msgr2=0x7f12dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:39.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.586+0000 7f12f7e00640 1 --2- 172.21.15.84:0/293149857 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f12dc0a3bb0 0x7f12dc0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f12f81a04e0 tx=0x7f12e0005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:39.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.586+0000 7f12f7e00640 1 -- 172.21.15.84:0/293149857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12f8108e70 msgr2=0x7f12f819efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:39.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.586+0000 7f12f7e00640 1 --2- 172.21.15.84:0/293149857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12f8108e70 0x7f12f819efd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f12e8005fd0 tx=0x7f12e800dde0 comp rx=0 tx=0).stop 2024-04-28T12:48:39.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.587+0000 7f12f7e00640 1 -- 172.21.15.84:0/293149857 shutdown_connections 2024-04-28T12:48:39.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.587+0000 7f12f7e00640 1 --2- 172.21.15.84:0/293149857 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f12dc0a3bb0 0x7f12dc0a6070 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f12f81a04e0 tx=0x7f12e0005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:39.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.587+0000 7f12f7e00640 1 --2- 172.21.15.84:0/293149857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12f810a980 0x7f12f81a6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:39.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.587+0000 7f12f7e00640 1 --2- 172.21.15.84:0/293149857 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12f8109fc0 0x7f12f819f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:39.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.587+0000 7f12f7e00640 1 --2- 172.21.15.84:0/293149857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12f8108e70 0x7f12f819efd0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f12e8005fd0 tx=0x7f12e800dde0 comp rx=0 tx=0).stop 2024-04-28T12:48:39.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.587+0000 7f12f7e00640 1 -- 172.21.15.84:0/293149857 >> 172.21.15.84:0/293149857 conn(0x7f12f8077e00 msgr2=0x7f12f810d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:39.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.590+0000 7f12f7e00640 1 -- 172.21.15.84:0/293149857 shutdown_connections 2024-04-28T12:48:39.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.590+0000 7f12f7e00640 1 -- 172.21.15.84:0/293149857 wait complete. 2024-04-28T12:48:39.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-28T12:48:39.793 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.797+0000 7f9955600640 1 -- 172.21.15.84:0/1948055922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99501028c0 msgr2=0x7f9950102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:39.793 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.797+0000 7f9955600640 1 --2- 172.21.15.84:0/1948055922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99501028c0 0x7f9950102ce0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9948009ca0 tx=0x7f994802a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:39.793 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.797+0000 7f9955600640 1 -- 172.21.15.84:0/1948055922 shutdown_connections 2024-04-28T12:48:39.793 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.797+0000 7f9955600640 1 --2- 172.21.15.84:0/1948055922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9950104470 0x7f995010ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:39.793 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.797+0000 7f9955600640 1 --2- 172.21.15.84:0/1948055922 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9950103ab0 0x7f9950103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:39.793 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.797+0000 7f9955600640 1 --2- 172.21.15.84:0/1948055922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99501028c0 0x7f9950102ce0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:39.793 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.797+0000 7f9955600640 1 -- 172.21.15.84:0/1948055922 >> 172.21.15.84:0/1948055922 conn(0x7f99500fde30 msgr2=0x7f9950100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:39.794 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.797+0000 7f9955600640 1 -- 172.21.15.84:0/1948055922 shutdown_connections 2024-04-28T12:48:39.794 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.797+0000 7f9955600640 1 -- 172.21.15.84:0/1948055922 wait complete. 2024-04-28T12:48:39.797 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.798+0000 7f9955600640 1 Processor -- start 2024-04-28T12:48:39.798 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.798+0000 7f9955600640 1 -- start start 2024-04-28T12:48:39.798 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.798+0000 7f9955600640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f99501028c0 0x7f995019ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:39.798 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.798+0000 7f9955600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9950103ab0 0x7f995019b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:39.798 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.798+0000 7f9955600640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9950104470 0x7f99501a2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:39.798 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.798+0000 7f9955600640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f995010e380 con 0x7f99501028c0 2024-04-28T12:48:39.798 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.798+0000 7f9955600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f995010e200 con 0x7f9950103ab0 2024-04-28T12:48:39.798 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.798+0000 7f9955600640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f995010e500 con 0x7f9950104470 2024-04-28T12:48:39.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.799+0000 7f994f400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9950103ab0 0x7f995019b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:39.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.799+0000 7f994fe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f99501028c0 0x7f995019ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:39.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.799+0000 7f994f400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9950103ab0 0x7f995019b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:39960/0 (socket says 172.21.15.84:39960) 2024-04-28T12:48:39.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.799+0000 7f994f400640 1 -- 172.21.15.84:0/2186735374 learned_addr learned my addr 172.21.15.84:0/2186735374 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:39.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.799+0000 7f9954c00640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9950104470 0x7f99501a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:39.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.799+0000 7f994f400640 1 -- 172.21.15.84:0/2186735374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9950104470 msgr2=0x7f99501a2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:39.799 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.799+0000 7f994f400640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9950104470 0x7f99501a2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:39.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.799+0000 7f994f400640 1 -- 172.21.15.84:0/2186735374 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f99501028c0 msgr2=0x7f995019ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:39.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.799+0000 7f994f400640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f99501028c0 0x7f995019ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:39.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.799+0000 7f994f400640 1 -- 172.21.15.84:0/2186735374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99501a28b0 con 0x7f9950103ab0 2024-04-28T12:48:39.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.800+0000 7f994f400640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9950103ab0 0x7f995019b1b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f993800ea60 tx=0x7f993800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:39.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.800+0000 7f994cc00640 1 -- 172.21.15.84:0/2186735374 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993800cde0 con 0x7f9950103ab0 2024-04-28T12:48:39.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.800+0000 7f994cc00640 1 -- 172.21.15.84:0/2186735374 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99380035f0 con 0x7f9950103ab0 2024-04-28T12:48:39.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.800+0000 7f9955600640 1 -- 172.21.15.84:0/2186735374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99501a2ba0 con 0x7f9950103ab0 2024-04-28T12:48:39.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.800+0000 7f994cc00640 1 -- 172.21.15.84:0/2186735374 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9938010690 con 0x7f9950103ab0 2024-04-28T12:48:39.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.800+0000 7f9955600640 1 -- 172.21.15.84:0/2186735374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99501a3140 con 0x7f9950103ab0 2024-04-28T12:48:39.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.801+0000 7f9955600640 1 -- 172.21.15.84:0/2186735374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99501a3370 con 0x7f9950103ab0 2024-04-28T12:48:39.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.803+0000 7f994cc00640 1 -- 172.21.15.84:0/2186735374 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f99380037a0 con 0x7f9950103ab0 2024-04-28T12:48:39.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.804+0000 7f994cc00640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f99340a3c30 0x7f99340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:39.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.805+0000 7f994cc00640 1 -- 172.21.15.84:0/2186735374 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7f99380d0ab0 con 0x7f9950103ab0 2024-04-28T12:48:39.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.805+0000 7f994fe00640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f99340a3c30 0x7f99340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:39.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.805+0000 7f994fe00640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f99340a3c30 0x7f99340a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9948006010 tx=0x7f9948000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:39.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:39.807+0000 7f994cc00640 1 -- 172.21.15.84:0/2186735374 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9938054060 con 0x7f9950103ab0 2024-04-28T12:48:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:39 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-28T12:48:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:39 smithi179 ceph-mon[63029]: osdmap e88: 12 total, 12 up, 12 in 2024-04-28T12:48:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:39 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/293149857' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-28T12:48:39.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:39 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-28T12:48:39.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:39 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-28T12:48:39.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:39 smithi084 ceph-mon[28081]: osdmap e88: 12 total, 12 up, 12 in 2024-04-28T12:48:39.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:39 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/293149857' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-28T12:48:39.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:39 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-28T12:48:39.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:39 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-28T12:48:39.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:39 smithi161 ceph-mon[62387]: osdmap e88: 12 total, 12 up, 12 in 2024-04-28T12:48:39.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:39 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/293149857' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-28T12:48:39.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:39 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-28T12:48:40.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.057+0000 7f9955600640 1 -- 172.21.15.84:0/2186735374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f99501a3540 con 0x7f9950103ab0 2024-04-28T12:48:40.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.588+0000 7f994cc00640 1 -- 172.21.15.84:0/2186735374 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f993809d210 con 0x7f9950103ab0 2024-04-28T12:48:40.584 INFO:teuthology.orchestra.run.smithi084.stderr:added data pool 4 to fsmap 2024-04-28T12:48:40.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.591+0000 7f9955600640 1 -- 172.21.15.84:0/2186735374 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f99340a3c30 msgr2=0x7f99340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:40.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.591+0000 7f9955600640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f99340a3c30 0x7f99340a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9948006010 tx=0x7f9948000f40 comp rx=0 tx=0).stop 2024-04-28T12:48:40.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.592+0000 7f9955600640 1 -- 172.21.15.84:0/2186735374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9950103ab0 msgr2=0x7f995019b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:40.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.592+0000 7f9955600640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9950103ab0 0x7f995019b1b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f993800ea60 tx=0x7f993800ef20 comp rx=0 tx=0).stop 2024-04-28T12:48:40.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.592+0000 7f9955600640 1 -- 172.21.15.84:0/2186735374 shutdown_connections 2024-04-28T12:48:40.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.592+0000 7f9955600640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f99340a3c30 0x7f99340a60f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.592+0000 7f9955600640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9950104470 0x7f99501a2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.592+0000 7f9955600640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9950103ab0 0x7f995019b1b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.592+0000 7f9955600640 1 --2- 172.21.15.84:0/2186735374 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f99501028c0 0x7f995019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.592+0000 7f9955600640 1 -- 172.21.15.84:0/2186735374 >> 172.21.15.84:0/2186735374 conn(0x7f99500fde30 msgr2=0x7f9950109250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:40.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.592+0000 7f9955600640 1 -- 172.21.15.84:0/2186735374 shutdown_connections 2024-04-28T12:48:40.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.592+0000 7f9955600640 1 -- 172.21.15.84:0/2186735374 wait complete. 2024-04-28T12:48:40.611 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:40.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.815+0000 7fac3c800640 1 -- 172.21.15.84:0/1468971153 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34106720 msgr2=0x7fac3410cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:40.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.815+0000 7fac3c800640 1 --2- 172.21.15.84:0/1468971153 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34106720 0x7fac3410cfb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fac30009ca0 tx=0x7fac3002a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.815+0000 7fac3c800640 1 -- 172.21.15.84:0/1468971153 shutdown_connections 2024-04-28T12:48:40.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.815+0000 7fac3c800640 1 --2- 172.21.15.84:0/1468971153 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34106720 0x7fac3410cfb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.815+0000 7fac3c800640 1 --2- 172.21.15.84:0/1468971153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34105d60 0x7fac341061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.815+0000 7fac3c800640 1 --2- 172.21.15.84:0/1468971153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34104b70 0x7fac34104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.815+0000 7fac3c800640 1 -- 172.21.15.84:0/1468971153 >> 172.21.15.84:0/1468971153 conn(0x7fac34100120 msgr2=0x7fac34102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:40.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.815+0000 7fac3c800640 1 -- 172.21.15.84:0/1468971153 shutdown_connections 2024-04-28T12:48:40.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.815+0000 7fac3c800640 1 -- 172.21.15.84:0/1468971153 wait complete. 2024-04-28T12:48:40.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.816+0000 7fac3c800640 1 Processor -- start 2024-04-28T12:48:40.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.817+0000 7fac3c800640 1 -- start start 2024-04-28T12:48:40.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.817+0000 7fac3c800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34104b70 0x7fac3419ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:40.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.817+0000 7fac3c800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34105d60 0x7fac3419b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:40.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.817+0000 7fac3c800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34106720 0x7fac341a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:40.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.817+0000 7fac3c800640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fac341106f0 con 0x7fac34106720 2024-04-28T12:48:40.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.817+0000 7fac3c800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fac34110570 con 0x7fac34104b70 2024-04-28T12:48:40.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.817+0000 7fac3c800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fac34110870 con 0x7fac34105d60 2024-04-28T12:48:40.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.817+0000 7fac39400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34105d60 0x7fac3419b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:40.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.817+0000 7fac39e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34104b70 0x7fac3419ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:40.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.817+0000 7fac3a800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34106720 0x7fac341a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:40.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.817+0000 7fac39400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34105d60 0x7fac3419b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:41948/0 (socket says 172.21.15.84:41948) 2024-04-28T12:48:40.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.817+0000 7fac39400640 1 -- 172.21.15.84:0/1721683230 learned_addr learned my addr 172.21.15.84:0/1721683230 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:40.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.818+0000 7fac39400640 1 -- 172.21.15.84:0/1721683230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34104b70 msgr2=0x7fac3419ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:40.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.818+0000 7fac39400640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34104b70 0x7fac3419ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.818+0000 7fac39400640 1 -- 172.21.15.84:0/1721683230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34106720 msgr2=0x7fac341a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:40.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.818+0000 7fac39400640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34106720 0x7fac341a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:40.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.818+0000 7fac39400640 1 -- 172.21.15.84:0/1721683230 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac341a2960 con 0x7fac34105d60 2024-04-28T12:48:40.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.818+0000 7fac39e00640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34104b70 0x7fac3419ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:40.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.818+0000 7fac39400640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34105d60 0x7fac3419b290 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fac2400e9f0 tx=0x7fac2400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:40.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.818+0000 7fac2ea00640 1 -- 172.21.15.84:0/1721683230 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2400cd80 con 0x7fac34105d60 2024-04-28T12:48:40.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.819+0000 7fac3c800640 1 -- 172.21.15.84:0/1721683230 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac341a2c50 con 0x7fac34105d60 2024-04-28T12:48:40.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.819+0000 7fac2ea00640 1 -- 172.21.15.84:0/1721683230 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac240054d0 con 0x7fac34105d60 2024-04-28T12:48:40.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.819+0000 7fac2ea00640 1 -- 172.21.15.84:0/1721683230 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac24010690 con 0x7fac34105d60 2024-04-28T12:48:40.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.819+0000 7fac3c800640 1 -- 172.21.15.84:0/1721683230 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac341a3150 con 0x7fac34105d60 2024-04-28T12:48:40.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.820+0000 7fac3c800640 1 -- 172.21.15.84:0/1721683230 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac3410be90 con 0x7fac34105d60 2024-04-28T12:48:40.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.822+0000 7fac2ea00640 1 -- 172.21.15.84:0/1721683230 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fac24004e90 con 0x7fac34105d60 2024-04-28T12:48:40.819 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.824+0000 7fac2ea00640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fac1c0a3bb0 0x7fac1c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:40.819 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.824+0000 7fac2ea00640 1 -- 172.21.15.84:0/1721683230 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fac240d0b00 con 0x7fac34105d60 2024-04-28T12:48:40.819 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.824+0000 7fac39e00640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fac1c0a3bb0 0x7fac1c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:40.819 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.824+0000 7fac39e00640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fac1c0a3bb0 0x7fac1c0a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fac3419c220 tx=0x7fac28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:40.821 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:40.825+0000 7fac2ea00640 1 -- 172.21.15.84:0/1721683230 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fac2409d240 con 0x7fac34105d60 2024-04-28T12:48:40.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:40 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-28T12:48:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:40 smithi179 ceph-mon[63029]: osdmap e89: 12 total, 12 up, 12 in 2024-04-28T12:48:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:40 smithi179 ceph-mon[63029]: pgmap v324: 129 pgs: 2 creating+peering, 97 active+clean, 30 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2024-04-28T12:48:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:40 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2186735374' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-28T12:48:40.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:40 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-28T12:48:40.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:40 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-28T12:48:40.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:40 smithi084 ceph-mon[28081]: osdmap e89: 12 total, 12 up, 12 in 2024-04-28T12:48:40.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:40 smithi084 ceph-mon[28081]: pgmap v324: 129 pgs: 2 creating+peering, 97 active+clean, 30 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2024-04-28T12:48:40.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:40 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2186735374' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-28T12:48:40.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:40 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-28T12:48:40.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:40 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-28T12:48:40.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:40 smithi161 ceph-mon[62387]: osdmap e89: 12 total, 12 up, 12 in 2024-04-28T12:48:40.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:40 smithi161 ceph-mon[62387]: pgmap v324: 129 pgs: 2 creating+peering, 97 active+clean, 30 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2024-04-28T12:48:40.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:40 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2186735374' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-28T12:48:40.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:40 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-28T12:48:41.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.065+0000 7fac3c800640 1 -- 172.21.15.84:0/1721683230 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac3402dad0 con 0x7fac34105d60 2024-04-28T12:48:41.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.066+0000 7fac2ea00640 1 -- 172.21.15.84:0/1721683230 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11812 (secure 0 0 0) 0x7fac2409cb40 con 0x7fac34105d60 2024-04-28T12:48:41.062 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:41.062 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6841/1196924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":1196924806},{"type":"v1","addr":"172.21.15.84:6841","nonce":1196924806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24562,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-28T12:48:34.334879+0000","modified":"2024-04-28T12:48:40.059686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24602},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6839/3812720642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":3812720642},{"type":"v1","addr":"172.21.15.161:6839","nonce":3812720642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24602,"qdb_cluster":[24602]},"id":1}]} 2024-04-28T12:48:41.062 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 17 2024-04-28T12:48:41.064 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.069+0000 7fac3c800640 1 -- 172.21.15.84:0/1721683230 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fac1c0a3bb0 msgr2=0x7fac1c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.064 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.069+0000 7fac3c800640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fac1c0a3bb0 0x7fac1c0a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fac3419c220 tx=0x7fac28005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:41.064 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.069+0000 7fac3c800640 1 -- 172.21.15.84:0/1721683230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34105d60 msgr2=0x7fac3419b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.069+0000 7fac3c800640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34105d60 0x7fac3419b290 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fac2400e9f0 tx=0x7fac2400edf0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.069+0000 7fac3c800640 1 -- 172.21.15.84:0/1721683230 shutdown_connections 2024-04-28T12:48:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.069+0000 7fac3c800640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fac1c0a3bb0 0x7fac1c0a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.069+0000 7fac3c800640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34106720 0x7fac341a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.069+0000 7fac3c800640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34105d60 0x7fac3419b290 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.069+0000 7fac3c800640 1 --2- 172.21.15.84:0/1721683230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34104b70 0x7fac3419ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.069+0000 7fac3c800640 1 -- 172.21.15.84:0/1721683230 >> 172.21.15.84:0/1721683230 conn(0x7fac34100120 msgr2=0x7fac34109060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.069+0000 7fac3c800640 1 -- 172.21.15.84:0/1721683230 shutdown_connections 2024-04-28T12:48:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.069+0000 7fac3c800640 1 -- 172.21.15.84:0/1721683230 wait complete. 2024-04-28T12:48:41.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:48:41.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.270+0000 7fed50a00640 1 -- 172.21.15.84:0/1372336392 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed4c104960 msgr2=0x7fed4c104d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.270+0000 7fed50a00640 1 --2- 172.21.15.84:0/1372336392 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed4c104960 0x7fed4c104d80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009900 tx=0x7fed3c02a3f0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.270+0000 7fed50a00640 1 -- 172.21.15.84:0/1372336392 shutdown_connections 2024-04-28T12:48:41.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.270+0000 7fed50a00640 1 --2- 172.21.15.84:0/1372336392 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed4c106510 0x7fed4c10cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.270+0000 7fed50a00640 1 --2- 172.21.15.84:0/1372336392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed4c105b50 0x7fed4c105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.270+0000 7fed50a00640 1 --2- 172.21.15.84:0/1372336392 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed4c104960 0x7fed4c104d80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.270+0000 7fed50a00640 1 -- 172.21.15.84:0/1372336392 >> 172.21.15.84:0/1372336392 conn(0x7fed4c0ffeb0 msgr2=0x7fed4c102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:41.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.270+0000 7fed50a00640 1 -- 172.21.15.84:0/1372336392 shutdown_connections 2024-04-28T12:48:41.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.270+0000 7fed50a00640 1 -- 172.21.15.84:0/1372336392 wait complete. 2024-04-28T12:48:41.267 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.272+0000 7fed50a00640 1 Processor -- start 2024-04-28T12:48:41.267 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.272+0000 7fed50a00640 1 -- start start 2024-04-28T12:48:41.269 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.272+0000 7fed50a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed4c104960 0x7fed4c19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.269 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.272+0000 7fed50a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed4c105b50 0x7fed4c19b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.269 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.272+0000 7fed50a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed4c106510 0x7fed4c1a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.269 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.272+0000 7fed50a00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c110490 con 0x7fed4c104960 2024-04-28T12:48:41.269 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.272+0000 7fed50a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c110310 con 0x7fed4c106510 2024-04-28T12:48:41.269 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.272+0000 7fed50a00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c110610 con 0x7fed4c105b50 2024-04-28T12:48:41.269 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.273+0000 7fed4a800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed4c104960 0x7fed4c19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.269 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.273+0000 7fed4a800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed4c104960 0x7fed4c19ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52800/0 (socket says 172.21.15.84:52800) 2024-04-28T12:48:41.269 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.273+0000 7fed4a800640 1 -- 172.21.15.84:0/4095857774 learned_addr learned my addr 172.21.15.84:0/4095857774 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:41.269 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.273+0000 7fed43e00640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed4c105b50 0x7fed4c19b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.273+0000 7fed4b200640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed4c106510 0x7fed4c1a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.273+0000 7fed4a800640 1 -- 172.21.15.84:0/4095857774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed4c105b50 msgr2=0x7fed4c19b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.273+0000 7fed4a800640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed4c105b50 0x7fed4c19b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.273+0000 7fed4a800640 1 -- 172.21.15.84:0/4095857774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed4c106510 msgr2=0x7fed4c1a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.273+0000 7fed4a800640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed4c106510 0x7fed4c1a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.273+0000 7fed4a800640 1 -- 172.21.15.84:0/4095857774 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4c1a27b0 con 0x7fed4c104960 2024-04-28T12:48:41.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.273+0000 7fed4a800640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed4c104960 0x7fed4c19ab90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fed3c006010 tx=0x7fed3c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:41.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.274+0000 7fed43400640 1 -- 172.21.15.84:0/4095857774 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c03f030 con 0x7fed4c104960 2024-04-28T12:48:41.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.274+0000 7fed43400640 1 -- 172.21.15.84:0/4095857774 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed3c031910 con 0x7fed4c104960 2024-04-28T12:48:41.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.274+0000 7fed50a00640 1 -- 172.21.15.84:0/4095857774 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4c1a2a40 con 0x7fed4c104960 2024-04-28T12:48:41.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.274+0000 7fed43400640 1 -- 172.21.15.84:0/4095857774 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c035070 con 0x7fed4c104960 2024-04-28T12:48:41.270 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.274+0000 7fed50a00640 1 -- 172.21.15.84:0/4095857774 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4c1a2f30 con 0x7fed4c104960 2024-04-28T12:48:41.271 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.275+0000 7fed50a00640 1 -- 172.21.15.84:0/4095857774 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed4c10bda0 con 0x7fed4c104960 2024-04-28T12:48:41.272 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.277+0000 7fed43400640 1 -- 172.21.15.84:0/4095857774 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fed3c042030 con 0x7fed4c104960 2024-04-28T12:48:41.275 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.280+0000 7fed43400640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed300a3bb0 0x7fed300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.275 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.280+0000 7fed43400640 1 -- 172.21.15.84:0/4095857774 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fed3c0f0f80 con 0x7fed4c104960 2024-04-28T12:48:41.275 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.280+0000 7fed43e00640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed300a3bb0 0x7fed300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.276 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.281+0000 7fed43e00640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed300a3bb0 0x7fed300a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fed4c19c0a0 tx=0x7fed34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:41.277 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.282+0000 7fed43400640 1 -- 172.21.15.84:0/4095857774 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fed3c0bd070 con 0x7fed4c104960 2024-04-28T12:48:41.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.476+0000 7fed50a00640 1 -- 172.21.15.84:0/4095857774 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed4c02dca0 con 0x7fed4c104960 2024-04-28T12:48:41.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.478+0000 7fed43400640 1 -- 172.21.15.84:0/4095857774 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24117 (secure 0 0 0) 0x7fed3c0c1370 con 0x7fed4c104960 2024-04-28T12:48:41.474 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:41.474 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":90,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:48:40.059255+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:48:41.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.480+0000 7fed50a00640 1 -- 172.21.15.84:0/4095857774 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed300a3bb0 msgr2=0x7fed300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.480+0000 7fed50a00640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed300a3bb0 0x7fed300a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fed4c19c0a0 tx=0x7fed34005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:41.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.481+0000 7fed50a00640 1 -- 172.21.15.84:0/4095857774 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed4c104960 msgr2=0x7fed4c19ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.481+0000 7fed50a00640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed4c104960 0x7fed4c19ab90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fed3c006010 tx=0x7fed3c0042f0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.481+0000 7fed50a00640 1 -- 172.21.15.84:0/4095857774 shutdown_connections 2024-04-28T12:48:41.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.481+0000 7fed50a00640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed300a3bb0 0x7fed300a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.481+0000 7fed50a00640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed4c106510 0x7fed4c1a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.481+0000 7fed50a00640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed4c105b50 0x7fed4c19b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.481+0000 7fed50a00640 1 --2- 172.21.15.84:0/4095857774 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed4c104960 0x7fed4c19ab90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.481+0000 7fed50a00640 1 -- 172.21.15.84:0/4095857774 >> 172.21.15.84:0/4095857774 conn(0x7fed4c0ffeb0 msgr2=0x7fed4c108f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:41.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.481+0000 7fed50a00640 1 -- 172.21.15.84:0/4095857774 shutdown_connections 2024-04-28T12:48:41.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.481+0000 7fed50a00640 1 -- 172.21.15.84:0/4095857774 wait complete. 2024-04-28T12:48:41.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:48:41.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.691+0000 7f4c2e200640 1 -- 172.21.15.84:0/20978416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c2810ef40 msgr2=0x7f4c28111330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.691+0000 7f4c2e200640 1 --2- 172.21.15.84:0/20978416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c2810ef40 0x7f4c28111330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4c20009cd0 tx=0x7f4c2002a8e0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.692+0000 7f4c2e200640 1 -- 172.21.15.84:0/20978416 shutdown_connections 2024-04-28T12:48:41.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.692+0000 7f4c2e200640 1 --2- 172.21.15.84:0/20978416 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c2810ef40 0x7f4c28111330 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.692+0000 7f4c2e200640 1 --2- 172.21.15.84:0/20978416 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c28075390 0x7f4c28075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.692+0000 7f4c2e200640 1 --2- 172.21.15.84:0/20978416 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4c28076970 0x7f4c28074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.692+0000 7f4c2e200640 1 -- 172.21.15.84:0/20978416 >> 172.21.15.84:0/20978416 conn(0x7f4c280fdee0 msgr2=0x7f4c28100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:41.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.692+0000 7f4c2e200640 1 -- 172.21.15.84:0/20978416 shutdown_connections 2024-04-28T12:48:41.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.692+0000 7f4c2e200640 1 -- 172.21.15.84:0/20978416 wait complete. 2024-04-28T12:48:41.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c2e200640 1 Processor -- start 2024-04-28T12:48:41.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c2e200640 1 -- start start 2024-04-28T12:48:41.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c2e200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c28075390 0x7f4c2819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c2e200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4c28076970 0x7f4c2819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c2e200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c2810ef40 0x7f4c281a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c2e200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4c28114a90 con 0x7f4c28075390 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c2e200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4c28114910 con 0x7f4c2810ef40 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c2e200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4c28114c10 con 0x7f4c28076970 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c27400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c28075390 0x7f4c2819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c27400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c28075390 0x7f4c2819f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52812/0 (socket says 172.21.15.84:52812) 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c27400640 1 -- 172.21.15.84:0/4252819804 learned_addr learned my addr 172.21.15.84:0/4252819804 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c27e00640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c2810ef40 0x7f4c281a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.693+0000 7f4c26a00640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4c28076970 0x7f4c2819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.694+0000 7f4c27400640 1 -- 172.21.15.84:0/4252819804 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4c28076970 msgr2=0x7f4c2819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.694+0000 7f4c27400640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4c28076970 0x7f4c2819f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.694+0000 7f4c27400640 1 -- 172.21.15.84:0/4252819804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c2810ef40 msgr2=0x7f4c281a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.694+0000 7f4c27400640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c2810ef40 0x7f4c281a66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.694+0000 7f4c27400640 1 -- 172.21.15.84:0/4252819804 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c281a6d60 con 0x7f4c28075390 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.694+0000 7f4c27400640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c28075390 0x7f4c2819f140 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800e9f0 tx=0x7f4c1800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:41.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.694+0000 7f4c1fe00640 1 -- 172.21.15.84:0/4252819804 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1800cd80 con 0x7f4c28075390 2024-04-28T12:48:41.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.694+0000 7f4c1fe00640 1 -- 172.21.15.84:0/4252819804 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c180054d0 con 0x7f4c28075390 2024-04-28T12:48:41.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.694+0000 7f4c2e200640 1 -- 172.21.15.84:0/4252819804 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c281a6ff0 con 0x7f4c28075390 2024-04-28T12:48:41.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.695+0000 7f4c1fe00640 1 -- 172.21.15.84:0/4252819804 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c18010690 con 0x7f4c28075390 2024-04-28T12:48:41.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.695+0000 7f4c2e200640 1 -- 172.21.15.84:0/4252819804 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c281a7460 con 0x7f4c28075390 2024-04-28T12:48:41.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.696+0000 7f4c2e200640 1 -- 172.21.15.84:0/4252819804 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c28105000 con 0x7f4c28075390 2024-04-28T12:48:41.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.698+0000 7f4c1fe00640 1 -- 172.21.15.84:0/4252819804 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4c18004e90 con 0x7f4c28075390 2024-04-28T12:48:41.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.699+0000 7f4c1fe00640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:41.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.699+0000 7f4c1fe00640 1 -- 172.21.15.84:0/4252819804 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4c180d0a80 con 0x7f4c28075390 2024-04-28T12:48:41.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.699+0000 7f4c26a00640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:41.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.700+0000 7f4c26a00640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4c140095f0 tx=0x7f4c14005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:41.699 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.704+0000 7f4c1fe00640 1 -- 172.21.15.84:0/4252819804 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4c180d2e20 con 0x7f4c28075390 2024-04-28T12:48:41.737 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:41 smithi084 ceph-mon[28081]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T12:48:41.737 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:41 smithi084 ceph-mon[28081]: osdmap e90: 12 total, 12 up, 12 in 2024-04-28T12:48:41.737 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:41 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-28T12:48:41.738 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:41 smithi084 ceph-mon[28081]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:48:41.738 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:41 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1721683230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.738 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:41 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4095857774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.914+0000 7f4c2e200640 1 -- 172.21.15.84:0/4252819804 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c280635d0 con 0x7f4c28075390 2024-04-28T12:48:41.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.917+0000 7f4c1fe00640 1 -- 172.21.15.84:0/4252819804 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24117 (secure 0 0 0) 0x7f4c1809d1c0 con 0x7f4c28075390 2024-04-28T12:48:41.913 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:41.913 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":90,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:48:40.059255+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:48:41.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:41 smithi179 ceph-mon[63029]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T12:48:41.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:41 smithi179 ceph-mon[63029]: osdmap e90: 12 total, 12 up, 12 in 2024-04-28T12:48:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:41 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-28T12:48:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:41 smithi179 ceph-mon[63029]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:48:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:41 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1721683230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:41 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4095857774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.920+0000 7f4c2e200640 1 -- 172.21.15.84:0/4252819804 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4c0c0a3bb0 msgr2=0x7f4c0c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.920+0000 7f4c2e200640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4c140095f0 tx=0x7f4c14005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:41.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.920+0000 7f4c2e200640 1 -- 172.21.15.84:0/4252819804 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c28075390 msgr2=0x7f4c2819f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:41.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.920+0000 7f4c2e200640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c28075390 0x7f4c2819f140 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800e9f0 tx=0x7f4c1800edf0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.920+0000 7f4c2e200640 1 -- 172.21.15.84:0/4252819804 shutdown_connections 2024-04-28T12:48:41.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.920+0000 7f4c2e200640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.920+0000 7f4c2e200640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c2810ef40 0x7f4c281a66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.920+0000 7f4c2e200640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4c28076970 0x7f4c2819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.920+0000 7f4c2e200640 1 --2- 172.21.15.84:0/4252819804 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c28075390 0x7f4c2819f140 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:41.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.920+0000 7f4c2e200640 1 -- 172.21.15.84:0/4252819804 >> 172.21.15.84:0/4252819804 conn(0x7f4c280fdee0 msgr2=0x7f4c281021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:41.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.920+0000 7f4c2e200640 1 -- 172.21.15.84:0/4252819804 shutdown_connections 2024-04-28T12:48:41.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:41.920+0000 7f4c2e200640 1 -- 172.21.15.84:0/4252819804 wait complete. 2024-04-28T12:48:41.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:48:41.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:41 smithi161 ceph-mon[62387]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-28T12:48:41.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:41 smithi161 ceph-mon[62387]: osdmap e90: 12 total, 12 up, 12 in 2024-04-28T12:48:41.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:41 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-28T12:48:41.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:41 smithi161 ceph-mon[62387]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:48:41.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:41 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1721683230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:41.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:41 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4095857774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:42.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.143+0000 7fbc2be00640 1 -- 172.21.15.84:0/320473387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c076930 msgr2=0x7fbc2c074d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.143+0000 7fbc2be00640 1 --2- 172.21.15.84:0/320473387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c076930 0x7fbc2c074d80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009860 tx=0x7fbc1c02a340 comp rx=0 tx=0).stop 2024-04-28T12:48:42.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.143+0000 7fbc2be00640 1 -- 172.21.15.84:0/320473387 shutdown_connections 2024-04-28T12:48:42.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.143+0000 7fbc2be00640 1 --2- 172.21.15.84:0/320473387 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc2c10a620 0x7fbc2c10ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.143+0000 7fbc2be00640 1 --2- 172.21.15.84:0/320473387 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc2c075350 0x7fbc2c0757d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.143+0000 7fbc2be00640 1 --2- 172.21.15.84:0/320473387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c076930 0x7fbc2c074d80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.143+0000 7fbc2be00640 1 -- 172.21.15.84:0/320473387 >> 172.21.15.84:0/320473387 conn(0x7fbc2c0fdcc0 msgr2=0x7fbc2c100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:42.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.143+0000 7fbc2be00640 1 -- 172.21.15.84:0/320473387 shutdown_connections 2024-04-28T12:48:42.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.143+0000 7fbc2be00640 1 -- 172.21.15.84:0/320473387 wait complete. 2024-04-28T12:48:42.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.144+0000 7fbc2be00640 1 Processor -- start 2024-04-28T12:48:42.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.145+0000 7fbc2be00640 1 -- start start 2024-04-28T12:48:42.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.145+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc2c075350 0x7fbc2c19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.145+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc2c076930 0x7fbc2c19b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.145+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c10a620 0x7fbc2c1a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.145+0000 7fbc2be00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c110480 con 0x7fbc2c075350 2024-04-28T12:48:42.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.145+0000 7fbc2be00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c110300 con 0x7fbc2c10a620 2024-04-28T12:48:42.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.145+0000 7fbc2be00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c110600 con 0x7fbc2c076930 2024-04-28T12:48:42.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.145+0000 7fbc2aa00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc2c075350 0x7fbc2c19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.145+0000 7fbc2aa00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc2c075350 0x7fbc2c19ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52824/0 (socket says 172.21.15.84:52824) 2024-04-28T12:48:42.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.145+0000 7fbc2aa00640 1 -- 172.21.15.84:0/559621500 learned_addr learned my addr 172.21.15.84:0/559621500 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:42.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.145+0000 7fbc2a000640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc2c076930 0x7fbc2c19b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.145+0000 7fbc2b400640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c10a620 0x7fbc2c1a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.142 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.146+0000 7fbc2aa00640 1 -- 172.21.15.84:0/559621500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc2c076930 msgr2=0x7fbc2c19b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.142 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.146+0000 7fbc2aa00640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc2c076930 0x7fbc2c19b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.142 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.146+0000 7fbc2aa00640 1 -- 172.21.15.84:0/559621500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c10a620 msgr2=0x7fbc2c1a2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.142 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.146+0000 7fbc2aa00640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c10a620 0x7fbc2c1a2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.146+0000 7fbc2aa00640 1 -- 172.21.15.84:0/559621500 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2c1a2790 con 0x7fbc2c075350 2024-04-28T12:48:42.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.146+0000 7fbc2aa00640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc2c075350 0x7fbc2c19ab70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0057c0 tx=0x7fbc1c032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:42.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.146+0000 7fbc23400640 1 -- 172.21.15.84:0/559621500 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c037040 con 0x7fbc2c075350 2024-04-28T12:48:42.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.146+0000 7fbc23400640 1 -- 172.21.15.84:0/559621500 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc1c030bd0 con 0x7fbc2c075350 2024-04-28T12:48:42.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.146+0000 7fbc2be00640 1 -- 172.21.15.84:0/559621500 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc2c1a2a20 con 0x7fbc2c075350 2024-04-28T12:48:42.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.146+0000 7fbc23400640 1 -- 172.21.15.84:0/559621500 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c005280 con 0x7fbc2c075350 2024-04-28T12:48:42.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.146+0000 7fbc2be00640 1 -- 172.21.15.84:0/559621500 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc2c1a2eb0 con 0x7fbc2c075350 2024-04-28T12:48:42.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.148+0000 7fbc2be00640 1 -- 172.21.15.84:0/559621500 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc2c1093a0 con 0x7fbc2c075350 2024-04-28T12:48:42.145 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.149+0000 7fbc23400640 1 -- 172.21.15.84:0/559621500 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbc1c043020 con 0x7fbc2c075350 2024-04-28T12:48:42.146 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.151+0000 7fbc23400640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbc100a3bb0 0x7fbc100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.146 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.151+0000 7fbc23400640 1 -- 172.21.15.84:0/559621500 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fbc1c0efd10 con 0x7fbc2c075350 2024-04-28T12:48:42.146 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.151+0000 7fbc2a000640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbc100a3bb0 0x7fbc100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.147 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.152+0000 7fbc2a000640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbc100a3bb0 0x7fbc100a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c19c080 tx=0x7fbc18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:42.150 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.155+0000 7fbc23400640 1 -- 172.21.15.84:0/559621500 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbc1c0f5020 con 0x7fbc2c075350 2024-04-28T12:48:42.393 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.398+0000 7fbc2be00640 1 -- 172.21.15.84:0/559621500 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc2c02dcf0 con 0x7fbc2c075350 2024-04-28T12:48:42.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.399+0000 7fbc23400640 1 -- 172.21.15.84:0/559621500 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11812 (secure 0 0 0) 0x7fbc1c04a020 con 0x7fbc2c075350 2024-04-28T12:48:42.395 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:42.395 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6841/1196924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":1196924806},{"type":"v1","addr":"172.21.15.84:6841","nonce":1196924806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24562,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-28T12:48:34.334879+0000","modified":"2024-04-28T12:48:40.059686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24602},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6839/3812720642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":3812720642},{"type":"v1","addr":"172.21.15.161:6839","nonce":3812720642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24602,"qdb_cluster":[24602]},"id":1}]} 2024-04-28T12:48:42.395 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 17 2024-04-28T12:48:42.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.402+0000 7fbc2be00640 1 -- 172.21.15.84:0/559621500 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbc100a3bb0 msgr2=0x7fbc100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.402+0000 7fbc2be00640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbc100a3bb0 0x7fbc100a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c19c080 tx=0x7fbc18005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:42.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.402+0000 7fbc2be00640 1 -- 172.21.15.84:0/559621500 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc2c075350 msgr2=0x7fbc2c19ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.402+0000 7fbc2be00640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc2c075350 0x7fbc2c19ab70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0057c0 tx=0x7fbc1c032da0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.402+0000 7fbc2be00640 1 -- 172.21.15.84:0/559621500 shutdown_connections 2024-04-28T12:48:42.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.402+0000 7fbc2be00640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbc100a3bb0 0x7fbc100a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.399 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.402+0000 7fbc2be00640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c10a620 0x7fbc2c1a2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.399 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.402+0000 7fbc2be00640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc2c076930 0x7fbc2c19b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.399 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.402+0000 7fbc2be00640 1 --2- 172.21.15.84:0/559621500 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbc2c075350 0x7fbc2c19ab70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.399 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.402+0000 7fbc2be00640 1 -- 172.21.15.84:0/559621500 >> 172.21.15.84:0/559621500 conn(0x7fbc2c0fdcc0 msgr2=0x7fbc2c106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:42.399 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.402+0000 7fbc2be00640 1 -- 172.21.15.84:0/559621500 shutdown_connections 2024-04-28T12:48:42.399 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.402+0000 7fbc2be00640 1 -- 172.21.15.84:0/559621500 wait complete. 2024-04-28T12:48:42.415 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:48:42.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.614+0000 7ff9d5000640 1 -- 172.21.15.84:0/2271194117 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d0103980 msgr2=0x7ff9d0103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.614+0000 7ff9d5000640 1 --2- 172.21.15.84:0/2271194117 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d0103980 0x7ff9d0103e00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009900 tx=0x7ff9bc02a3f0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.614+0000 7ff9d5000640 1 -- 172.21.15.84:0/2271194117 shutdown_connections 2024-04-28T12:48:42.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.614+0000 7ff9d5000640 1 --2- 172.21.15.84:0/2271194117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d0104340 0x7ff9d010abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.614+0000 7ff9d5000640 1 --2- 172.21.15.84:0/2271194117 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d0103980 0x7ff9d0103e00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.614+0000 7ff9d5000640 1 --2- 172.21.15.84:0/2271194117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9d0102790 0x7ff9d0102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.614+0000 7ff9d5000640 1 -- 172.21.15.84:0/2271194117 >> 172.21.15.84:0/2271194117 conn(0x7ff9d00fdd00 msgr2=0x7ff9d0100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:42.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.614+0000 7ff9d5000640 1 -- 172.21.15.84:0/2271194117 shutdown_connections 2024-04-28T12:48:42.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.614+0000 7ff9d5000640 1 -- 172.21.15.84:0/2271194117 wait complete. 2024-04-28T12:48:42.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.615+0000 7ff9d5000640 1 Processor -- start 2024-04-28T12:48:42.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.615+0000 7ff9d5000640 1 -- start start 2024-04-28T12:48:42.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.615+0000 7ff9d5000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d0102790 0x7ff9d019ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.615+0000 7ff9d5000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d0103980 0x7ff9d019b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.615+0000 7ff9d5000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9d0104340 0x7ff9d01a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.615+0000 7ff9d5000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d010e2a0 con 0x7ff9d0102790 2024-04-28T12:48:42.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.615+0000 7ff9d5000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d010e120 con 0x7ff9d0103980 2024-04-28T12:48:42.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.615+0000 7ff9d5000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d010e420 con 0x7ff9d0104340 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.616+0000 7ff9cf400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d0102790 0x7ff9d019ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.616+0000 7ff9cf400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d0102790 0x7ff9d019ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52842/0 (socket says 172.21.15.84:52842) 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.616+0000 7ff9cf400640 1 -- 172.21.15.84:0/2119986788 learned_addr learned my addr 172.21.15.84:0/2119986788 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.616+0000 7ff9cea00640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d0103980 0x7ff9d019b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.616+0000 7ff9cfe00640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9d0104340 0x7ff9d01a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.616+0000 7ff9cf400640 1 -- 172.21.15.84:0/2119986788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9d0104340 msgr2=0x7ff9d01a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.616+0000 7ff9cf400640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9d0104340 0x7ff9d01a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.616+0000 7ff9cf400640 1 -- 172.21.15.84:0/2119986788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d0103980 msgr2=0x7ff9d019b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.616+0000 7ff9cf400640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d0103980 0x7ff9d019b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.616+0000 7ff9cf400640 1 -- 172.21.15.84:0/2119986788 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d01a27b0 con 0x7ff9d0102790 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.616+0000 7ff9cf400640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d0102790 0x7ff9d019ab90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000d770 tx=0x7ff9c000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.617+0000 7ff9c7e00640 1 -- 172.21.15.84:0/2119986788 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0017070 con 0x7ff9d0102790 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.617+0000 7ff9c7e00640 1 -- 172.21.15.84:0/2119986788 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9c0009c30 con 0x7ff9d0102790 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.617+0000 7ff9d5000640 1 -- 172.21.15.84:0/2119986788 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d01a2aa0 con 0x7ff9d0102790 2024-04-28T12:48:42.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.617+0000 7ff9c7e00640 1 -- 172.21.15.84:0/2119986788 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0003cc0 con 0x7ff9d0102790 2024-04-28T12:48:42.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.617+0000 7ff9d5000640 1 -- 172.21.15.84:0/2119986788 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d01a2fc0 con 0x7ff9d0102790 2024-04-28T12:48:42.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.618+0000 7ff9d5000640 1 -- 172.21.15.84:0/2119986788 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d01a31f0 con 0x7ff9d0102790 2024-04-28T12:48:42.615 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.620+0000 7ff9c7e00640 1 -- 172.21.15.84:0/2119986788 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff9c0003e60 con 0x7ff9d0102790 2024-04-28T12:48:42.616 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.621+0000 7ff9c7e00640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff9b40a3bb0 0x7ff9b40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:42.617 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.622+0000 7ff9c7e00640 1 -- 172.21.15.84:0/2119986788 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff9c00cfff0 con 0x7ff9d0102790 2024-04-28T12:48:42.617 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.622+0000 7ff9cea00640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff9b40a3bb0 0x7ff9b40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:42.617 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.622+0000 7ff9cea00640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff9b40a3bb0 0x7ff9b40a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc000c00 tx=0x7ff9bc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:42.620 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.625+0000 7ff9c7e00640 1 -- 172.21.15.84:0/2119986788 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff9c009c770 con 0x7ff9d0102790 2024-04-28T12:48:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:42 smithi084 ceph-mon[28081]: pgmap v326: 129 pgs: 1 creating+activating, 2 creating+peering, 118 active+clean, 8 unknown; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2024-04-28T12:48:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:42 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4252819804' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:42.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:42 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/559621500' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:42.823 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.827+0000 7ff9d5000640 1 -- 172.21.15.84:0/2119986788 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9d01a3a00 con 0x7ff9d0102790 2024-04-28T12:48:42.825 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.830+0000 7ff9c7e00640 1 -- 172.21.15.84:0/2119986788 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24117 (secure 0 0 0) 0x7ff9c0009de0 con 0x7ff9d0102790 2024-04-28T12:48:42.826 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:42.826 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":90,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:48:40.059255+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:48:42.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.832+0000 7ff9d5000640 1 -- 172.21.15.84:0/2119986788 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff9b40a3bb0 msgr2=0x7ff9b40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.832+0000 7ff9d5000640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff9b40a3bb0 0x7ff9b40a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc000c00 tx=0x7ff9bc0314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.832+0000 7ff9d5000640 1 -- 172.21.15.84:0/2119986788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d0102790 msgr2=0x7ff9d019ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:42.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.832+0000 7ff9d5000640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d0102790 0x7ff9d019ab90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000d770 tx=0x7ff9c000dc30 comp rx=0 tx=0).stop 2024-04-28T12:48:42.829 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.833+0000 7ff9d5000640 1 -- 172.21.15.84:0/2119986788 shutdown_connections 2024-04-28T12:48:42.829 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.833+0000 7ff9d5000640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff9b40a3bb0 0x7ff9b40a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.829 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.833+0000 7ff9d5000640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9d0104340 0x7ff9d01a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.829 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.833+0000 7ff9d5000640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d0103980 0x7ff9d019b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.829 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.833+0000 7ff9d5000640 1 --2- 172.21.15.84:0/2119986788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d0102790 0x7ff9d019ab90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:42.829 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.833+0000 7ff9d5000640 1 -- 172.21.15.84:0/2119986788 >> 172.21.15.84:0/2119986788 conn(0x7ff9d00fdd00 msgr2=0x7ff9d0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:42.829 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.833+0000 7ff9d5000640 1 -- 172.21.15.84:0/2119986788 shutdown_connections 2024-04-28T12:48:42.829 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:42.833+0000 7ff9d5000640 1 -- 172.21.15.84:0/2119986788 wait complete. 2024-04-28T12:48:42.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-28T12:48:34.334879+0000', 'modified': '2024-04-28T12:48:40.059686+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24602}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24602': {'gid': 24602, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6839/3812720642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 3812720642}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 3812720642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24602, 'qdb_cluster': [24602]} 2024-04-28T12:48:42.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-28T12:48:42.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-28T12:48:42.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:42 smithi179 ceph-mon[63029]: pgmap v326: 129 pgs: 1 creating+activating, 2 creating+peering, 118 active+clean, 8 unknown; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2024-04-28T12:48:42.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:42 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4252819804' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:42.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:42 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/559621500' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:42.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:42 smithi161 ceph-mon[62387]: pgmap v326: 129 pgs: 1 creating+activating, 2 creating+peering, 118 active+clean, 8 unknown; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2024-04-28T12:48:42.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:42 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4252819804' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:42.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:42 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/559621500' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:48:43.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.047+0000 7fe0f9000640 1 -- 172.21.15.84:0/3422528607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0f40772f0 msgr2=0x7fe0f4077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.047+0000 7fe0f9000640 1 --2- 172.21.15.84:0/3422528607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0f40772f0 0x7fe0f4077710 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009900 tx=0x7fe0e402a410 comp rx=0 tx=0).stop 2024-04-28T12:48:43.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.047+0000 7fe0f9000640 1 -- 172.21.15.84:0/3422528607 shutdown_connections 2024-04-28T12:48:43.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.047+0000 7fe0f9000640 1 --2- 172.21.15.84:0/3422528607 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0f4078ed0 0x7fe0f4079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.047+0000 7fe0f9000640 1 --2- 172.21.15.84:0/3422528607 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f4078530 0x7fe0f4078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.047+0000 7fe0f9000640 1 --2- 172.21.15.84:0/3422528607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0f40772f0 0x7fe0f4077710 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.047+0000 7fe0f9000640 1 -- 172.21.15.84:0/3422528607 >> 172.21.15.84:0/3422528607 conn(0x7fe0f41000f0 msgr2=0x7fe0f4102530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:43.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.047+0000 7fe0f9000640 1 -- 172.21.15.84:0/3422528607 shutdown_connections 2024-04-28T12:48:43.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.047+0000 7fe0f9000640 1 -- 172.21.15.84:0/3422528607 wait complete. 2024-04-28T12:48:43.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.048+0000 7fe0f9000640 1 Processor -- start 2024-04-28T12:48:43.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.048+0000 7fe0f9000640 1 -- start start 2024-04-28T12:48:43.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.048+0000 7fe0f9000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0f40772f0 0x7fe0f419f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.048+0000 7fe0f9000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0f4078530 0x7fe0f419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.048+0000 7fe0f9000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f4078ed0 0x7fe0f41a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.048+0000 7fe0f9000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f4114ac0 con 0x7fe0f4078530 2024-04-28T12:48:43.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.048+0000 7fe0f9000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f4114940 con 0x7fe0f40772f0 2024-04-28T12:48:43.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.048+0000 7fe0f9000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f4114c40 con 0x7fe0f4078ed0 2024-04-28T12:48:43.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.049+0000 7fe0ebe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0f4078530 0x7fe0f419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.049+0000 7fe0ebe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0f4078530 0x7fe0f419f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52860/0 (socket says 172.21.15.84:52860) 2024-04-28T12:48:43.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.049+0000 7fe0ebe00640 1 -- 172.21.15.84:0/3055993542 learned_addr learned my addr 172.21.15.84:0/3055993542 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.049+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0f40772f0 0x7fe0f419f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.049+0000 7fe0f2800640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f4078ed0 0x7fe0f41a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.049+0000 7fe0ebe00640 1 -- 172.21.15.84:0/3055993542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f4078ed0 msgr2=0x7fe0f41a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.049+0000 7fe0ebe00640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f4078ed0 0x7fe0f41a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.049+0000 7fe0ebe00640 1 -- 172.21.15.84:0/3055993542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0f40772f0 msgr2=0x7fe0f419f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.049+0000 7fe0ebe00640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0f40772f0 0x7fe0f419f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.049+0000 7fe0ebe00640 1 -- 172.21.15.84:0/3055993542 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f41a6da0 con 0x7fe0f4078530 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.049+0000 7fe0f2800640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f4078ed0 0x7fe0f41a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.049+0000 7fe0ebe00640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0f4078530 0x7fe0f419f6c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00e9f0 tx=0x7fe0dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.050+0000 7fe0eaa00640 1 -- 172.21.15.84:0/3055993542 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc00cd80 con 0x7fe0f4078530 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.050+0000 7fe0eaa00640 1 -- 172.21.15.84:0/3055993542 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0dc0108e0 con 0x7fe0f4078530 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.050+0000 7fe0f9000640 1 -- 172.21.15.84:0/3055993542 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0f41a7090 con 0x7fe0f4078530 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.050+0000 7fe0eaa00640 1 -- 172.21.15.84:0/3055993542 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc018980 con 0x7fe0f4078530 2024-04-28T12:48:43.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.050+0000 7fe0f9000640 1 -- 172.21.15.84:0/3055993542 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0f41a7630 con 0x7fe0f4078530 2024-04-28T12:48:43.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.053+0000 7fe0fbe4d740 1 Processor -- start 2024-04-28T12:48:43.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.053+0000 7fe0fbe4d740 1 -- start start 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.053+0000 7fe0eaa00640 1 -- 172.21.15.84:0/3055993542 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe0dc010430 con 0x7fe0f4078530 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.053+0000 7fe0fbe4d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd296d0f0 0x562cd296d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.053+0000 7fe0fbe4d740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd29725e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.053+0000 7fe0fbe4d740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd2972c50 0x562cd29750e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.053+0000 7fe0fbe4d740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x562cd28d1600 con 0x562cd296dae0 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.053+0000 7fe0fbe4d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x562cd286e100 con 0x562cd296d0f0 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.053+0000 7fe0fbe4d740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562cd28cd5c0 con 0x562cd2972c50 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.053+0000 7fe0f2800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd296d0f0 0x562cd296d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0f1e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd29725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0f2800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd296d0f0 0x562cd296d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:40058/0 (socket says 172.21.15.84:40058) 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0f2800640 1 -- 172.21.15.84:0/3042712 learned_addr learned my addr 172.21.15.84:0/3042712 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0ebe00640 1 --2- 172.21.15.84:0/3042712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd2972c50 0x562cd29750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0f2800640 1 -- 172.21.15.84:0/3042712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd2972c50 msgr2=0x562cd29750e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:48:43.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0f2800640 1 --2- 172.21.15.84:0/3042712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd2972c50 0x562cd29750e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0f2800640 1 -- 172.21.15.84:0/3042712 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 msgr2=0x562cd29725e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:48:43.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0f2800640 1 --2- 172.21.15.84:0/3042712 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd29725e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0f2800640 1 -- 172.21.15.84:0/3042712 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cd25e32d0 con 0x562cd296d0f0 2024-04-28T12:48:43.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3042712 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd29725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:43.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0f2800640 1 --2- 172.21.15.84:0/3042712 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd296d0f0 0x562cd296d510 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009fd0 tx=0x7fe0ec0108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fef90cb525a6cec server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:43.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0e3400640 1 -- 172.21.15.84:0/3042712 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec011760 con 0x562cd296d0f0 2024-04-28T12:48:43.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.054+0000 7fe0e3400640 1 -- 172.21.15.84:0/3042712 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ec011db0 con 0x562cd296d0f0 2024-04-28T12:48:43.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.055+0000 7fe0e3400640 1 -- 172.21.15.84:0/3042712 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec00e3c0 con 0x562cd296d0f0 2024-04-28T12:48:43.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.055+0000 7fe0eaa00640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe0d80a3c30 0x7fe0d80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.055+0000 7fe0eaa00640 1 -- 172.21.15.84:0/3055993542 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe0dc0d09b0 con 0x7fe0f4078530 2024-04-28T12:48:43.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.055+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe0d80a3c30 0x7fe0d80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.055+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3042712 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd296d0f0 msgr2=0x562cd296d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.055+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3042712 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd296d0f0 0x562cd296d510 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009fd0 tx=0x7fe0ec0108e0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.056+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3042712 shutdown_connections 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.056+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe0d80a3c30 0x7fe0d80a60f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4000c00 tx=0x7fe0e40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.056+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3042712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd2972c50 0x562cd29750e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.056+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3042712 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd29725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.056+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3042712 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd296d0f0 0x562cd296d510 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.056+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3042712 >> 172.21.15.84:0/3042712 conn(0x562cd2901d10 msgr2=0x562cd2968b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.056+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3042712 shutdown_connections 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.056+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3042712 wait complete. 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.056+0000 7fe0fbe4d740 1 -- start start 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.056+0000 7fe0fbe4d740 1 Processor -- start 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0fbe4d740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd296d0f0 0x562cd2977570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0fbe4d740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd2977ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0fbe4d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd2972c50 0x562cd297eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0fbe4d740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x562cd2975c00 con 0x562cd296dae0 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0fbe4d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x562cd2858f70 con 0x562cd2972c50 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0fbe4d740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562cd2975d80 con 0x562cd296d0f0 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0fbe4d740 10 client.0 _ll_register_callbacks cb 0x562cd28c7480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0ebe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd2977ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0ebe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd2977ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52876/0 (socket says 172.21.15.84:52876) 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0ebe00640 1 -- 172.21.15.84:0/3490737618 learned_addr learned my addr 172.21.15.84:0/3490737618 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd296d0f0 0x562cd2977570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0f2800640 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd2972c50 0x562cd297eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0ebe00640 1 -- 172.21.15.84:0/3490737618 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd296d0f0 msgr2=0x562cd2977570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0ebe00640 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd296d0f0 0x562cd2977570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0ebe00640 1 -- 172.21.15.84:0/3490737618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd2972c50 msgr2=0x562cd297eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0ebe00640 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd2972c50 0x562cd297eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0ebe00640 1 -- 172.21.15.84:0/3490737618 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cd26ad6b0 con 0x562cd296dae0 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.057+0000 7fe0f2800640 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd2972c50 0x562cd297eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.058+0000 7fe0ebe00640 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd2977ab0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc099420 tx=0x7fe0dc00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae4221383654527 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.058+0000 7fe0d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.84:3300/0 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.058+0000 7fe0d7e00640 1 -- 172.21.15.84:0/3490737618 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc0d0e60 con 0x562cd296dae0 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.058+0000 7fe0d7e00640 1 -- 172.21.15.84:0/3490737618 <== mon.0 v2:172.21.15.84:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe0dc010a90 con 0x562cd296dae0 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.058+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3490737618 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0ec011db0 con 0x562cd296dae0 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.058+0000 7fe0d7e00640 1 -- 172.21.15.84:0/3490737618 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fe0dc09d150 con 0x562cd296dae0 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.058+0000 7fe0fbe4d740 10 client.14583 fetch_fsmap learned FSMap version 17 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.058+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3490737618 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x562cd2981b40 con 0x562cd296dae0 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.058+0000 7fe0d7e00640 1 -- 172.21.15.84:0/3490737618 <== mon.0 v2:172.21.15.84:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ec011db0 con 0x562cd296dae0 2024-04-28T12:48:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.058+0000 7fe0d7e00640 1 -- 172.21.15.84:0/3490737618 <== mon.0 v2:172.21.15.84:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fe0dc0987a0 con 0x562cd296dae0 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.058+0000 7fe0fbe4d740 10 client.14583 fetch_fsmap finished waiting for FSMap version 17 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.059+0000 7fe0fbe4d740 10 client.14583 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.059+0000 7fe0fbe4d740 20 client.14583 populate_metadata read hostname 'smithi084' 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.059+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x562cd2984180 0x562cd29855f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.059+0000 7fe0fbe4d740 4 client.14583 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.059+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3490737618 --> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562cd28df210 con 0x562cd2984180 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.059+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x562cd2984180 0x562cd29855f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.060+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x562cd2984180 0x562cd29855f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41ddcbf6897d62ea server_cookie=989b43c210904428 in_seq=0 out_seq=0 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.060+0000 7fe0d7e00640 10 client.14583 ms_handle_connect on v2:172.21.15.161:6838/3812720642 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0d7e00640 1 -- 172.21.15.84:0/3490737618 <== mds.0 v2:172.21.15.161:6838/3812720642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562cd28df210 con 0x562cd2984180 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0d7e00640 10 client.14583 handle_command_reply: tid=0 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0fbe4d740 1 client.14583 shutdown 2024-04-28T12:48:43.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3490737618 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 msgr2=0x562cd2977ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd2977ab0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc099420 tx=0x7fe0dc00cf20 comp rx=0 tx=0).stop 2024-04-28T12:48:43.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3490737618 shutdown_connections 2024-04-28T12:48:43.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x562cd2984180 0x562cd29855f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd2972c50 0x562cd297eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0d7e00640 0 client.14583 ms_handle_reset on v2:172.21.15.161:6838/3812720642 2024-04-28T12:48:43.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd2977ab0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3490737618 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd296d0f0 0x562cd2977570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3490737618 >> 172.21.15.84:0/3490737618 conn(0x562cd2901d10 msgr2=0x562cd2967eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:43.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3490737618 shutdown_connections 2024-04-28T12:48:43.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3490737618 wait complete. 2024-04-28T12:48:43.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.061+0000 7fe0fbe4d740 20 client.14583 trim_cache size 0 max 16384 2024-04-28T12:48:43.077 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0fbe4d740 1 Processor -- start 2024-04-28T12:48:43.077 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0fbe4d740 1 -- start start 2024-04-28T12:48:43.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0fbe4d740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296d0f0 0x562cd296ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0fbe4d740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd296dae0 0x562cd296f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0fbe4d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd2972c50 0x562cd297eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0fbe4d740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x562cd2983930 con 0x562cd296d0f0 2024-04-28T12:48:43.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0fbe4d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x562cd2984000 con 0x562cd2972c50 2024-04-28T12:48:43.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0fbe4d740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562cd29846d0 con 0x562cd296dae0 2024-04-28T12:48:43.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0f1e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296d0f0 0x562cd296ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0f1e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296d0f0 0x562cd296ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52890/0 (socket says 172.21.15.84:52890) 2024-04-28T12:48:43.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0f1e00640 1 -- 172.21.15.84:0/3274464880 learned_addr learned my addr 172.21.15.84:0/3274464880 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:43.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0ebe00640 1 --2- 172.21.15.84:0/3274464880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd296dae0 0x562cd296f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.081+0000 7fe0f2800640 1 --2- 172.21.15.84:0/3274464880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd2972c50 0x562cd297eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.082+0000 7fe0f1e00640 1 -- 172.21.15.84:0/3274464880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd296dae0 msgr2=0x562cd296f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:48:43.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.082+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3274464880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd296dae0 0x562cd296f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.082+0000 7fe0f1e00640 1 -- 172.21.15.84:0/3274464880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd2972c50 msgr2=0x562cd297eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:48:43.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.082+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3274464880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd2972c50 0x562cd297eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.082+0000 7fe0f1e00640 1 -- 172.21.15.84:0/3274464880 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cd2972220 con 0x562cd296d0f0 2024-04-28T12:48:43.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.082+0000 7fe0ebe00640 1 --2- 172.21.15.84:0/3274464880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd296dae0 0x562cd296f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:43.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.082+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3274464880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296d0f0 0x562cd296ef50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009900 tx=0x7fe0e4040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93802b5f39759e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:43.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.082+0000 7fe0e2a00640 1 -- 172.21.15.84:0/3274464880 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4035040 con 0x562cd296d0f0 2024-04-28T12:48:43.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.082+0000 7fe0e2a00640 1 -- 172.21.15.84:0/3274464880 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0e4004040 con 0x562cd296d0f0 2024-04-28T12:48:43.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.082+0000 7fe0e2a00640 1 -- 172.21.15.84:0/3274464880 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4051850 con 0x562cd296d0f0 2024-04-28T12:48:43.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.083+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3274464880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296d0f0 msgr2=0x562cd296ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.083+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3274464880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296d0f0 0x562cd296ef50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009900 tx=0x7fe0e4040000 comp rx=0 tx=0).stop 2024-04-28T12:48:43.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.083+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3274464880 shutdown_connections 2024-04-28T12:48:43.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.083+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3274464880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd2972c50 0x562cd297eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.083+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3274464880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd296dae0 0x562cd296f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.083+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3274464880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296d0f0 0x562cd296ef50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.083+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3274464880 >> 172.21.15.84:0/3274464880 conn(0x562cd2901d10 msgr2=0x562cd2977570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.083+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3274464880 shutdown_connections 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.083+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3274464880 wait complete. 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.083+0000 7fe0fbe4d740 1 -- start start 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.083+0000 7fe0fbe4d740 1 Processor -- start 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0fbe4d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd296d0f0 0x562cd297e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0fbe4d740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd297b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0fbe4d740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd2972c50 0x562cd297b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0fbe4d740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x562cd2985470 con 0x562cd296dae0 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0fbe4d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x562cd2984da0 con 0x562cd296d0f0 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0fbe4d740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x562cd2985b40 con 0x562cd2972c50 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0fbe4d740 10 client.0 _ll_register_callbacks cb 0x7fe0dc009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0ebe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd297b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0f2800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd2972c50 0x562cd297b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0ebe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd297b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52892/0 (socket says 172.21.15.84:52892) 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0ebe00640 1 -- 172.21.15.84:0/3535401316 learned_addr learned my addr 172.21.15.84:0/3535401316 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3535401316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd296d0f0 0x562cd297e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0ebe00640 1 -- 172.21.15.84:0/3535401316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd2972c50 msgr2=0x562cd297b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0ebe00640 1 --2- 172.21.15.84:0/3535401316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd2972c50 0x562cd297b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0ebe00640 1 -- 172.21.15.84:0/3535401316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd296d0f0 msgr2=0x562cd297e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0ebe00640 1 --2- 172.21.15.84:0/3535401316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd296d0f0 0x562cd297e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.084+0000 7fe0ebe00640 1 -- 172.21.15.84:0/3535401316 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cd297ecd0 con 0x562cd296dae0 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.085+0000 7fe0ebe00640 1 --2- 172.21.15.84:0/3535401316 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd297b140 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc09c750 tx=0x7fe0dc010720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3fc3bbdf9b3cce server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.085+0000 7fe0d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.84:3300/0 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.085+0000 7fe0d7e00640 1 -- 172.21.15.84:0/3535401316 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc09c590 con 0x562cd296dae0 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.085+0000 7fe0d7e00640 1 -- 172.21.15.84:0/3535401316 <== mon.0 v2:172.21.15.84:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe0dc09bea0 con 0x562cd296dae0 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.085+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3535401316 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0e4004040 con 0x562cd296dae0 2024-04-28T12:48:43.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.085+0000 7fe0d7e00640 1 -- 172.21.15.84:0/3535401316 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fe0dc09ca50 con 0x562cd296dae0 2024-04-28T12:48:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.085+0000 7fe0fbe4d740 10 client.14589 fetch_fsmap learned FSMap version 17 2024-04-28T12:48:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.085+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3535401316 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x562cd29eb040 con 0x562cd296dae0 2024-04-28T12:48:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.085+0000 7fe0d7e00640 1 -- 172.21.15.84:0/3535401316 <== mon.0 v2:172.21.15.84:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0e4004040 con 0x562cd296dae0 2024-04-28T12:48:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.085+0000 7fe0d7e00640 1 -- 172.21.15.84:0/3535401316 <== mon.0 v2:172.21.15.84:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fe0dc09b9a0 con 0x562cd296dae0 2024-04-28T12:48:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.086+0000 7fe0fbe4d740 10 client.14589 fetch_fsmap finished waiting for FSMap version 17 2024-04-28T12:48:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.086+0000 7fe0fbe4d740 10 client.14589 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-28T12:48:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.086+0000 7fe0fbe4d740 20 client.14589 populate_metadata read hostname 'smithi084' 2024-04-28T12:48:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.086+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3535401316 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x562cd29efc20 0x562cd29ee930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.086+0000 7fe0fbe4d740 4 client.14589 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:48:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.086+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3535401316 --> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] -- command(tid 0: {"prefix": "status"}) -- 0x562cd28d9080 con 0x562cd29efc20 2024-04-28T12:48:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.086+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3535401316 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x562cd29efc20 0x562cd29ee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:43.086 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0f1e00640 1 --2- 172.21.15.84:0/3535401316 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x562cd29efc20 0x562cd29ee930 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54689987ecce567e server_cookie=e2a313457603dd33 in_seq=0 out_seq=0 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0d7e00640 10 client.14589 ms_handle_connect on v2:172.21.15.161:6838/3812720642 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0d7e00640 1 -- 172.21.15.84:0/3535401316 <== mds.0 v2:172.21.15.161:6838/3812720642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x562cd28d9080 con 0x562cd29efc20 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0d7e00640 10 client.14589 handle_command_reply: tid=0 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0fbe4d740 1 client.14589 shutdown 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3535401316 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 msgr2=0x562cd297b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3535401316 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd297b140 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc09c750 tx=0x7fe0dc010720 comp rx=0 tx=0).stop 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3535401316 shutdown_connections 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3535401316 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x562cd29efc20 0x562cd29ee930 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3535401316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562cd2972c50 0x562cd297b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3535401316 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x562cd296dae0 0x562cd297b140 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0fbe4d740 1 --2- 172.21.15.84:0/3535401316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cd296d0f0 0x562cd297e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0d7e00640 0 client.14589 ms_handle_reset on v2:172.21.15.161:6838/3812720642 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3535401316 >> 172.21.15.84:0/3535401316 conn(0x562cd2901d10 msgr2=0x562cd2977200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:43.087 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.087+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3535401316 shutdown_connections 2024-04-28T12:48:43.088 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0fbe4d740 1 -- 172.21.15.84:0/3535401316 wait complete. 2024-04-28T12:48:43.088 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0fbe4d740 20 client.14589 trim_cache size 0 max 16384 2024-04-28T12:48:43.088 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-28T12:48:43.088 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_fsid": "ddee604a-055b-11ef-bc93-c7b262605968", 2024-04-28T12:48:43.088 INFO:teuthology.orchestra.run.smithi084.stdout: "whoami": 0, 2024-04-28T12:48:43.088 INFO:teuthology.orchestra.run.smithi084.stdout: "id": 24602, 2024-04-28T12:48:43.088 INFO:teuthology.orchestra.run.smithi084.stdout: "want_state": "up:active", 2024-04-28T12:48:43.088 INFO:teuthology.orchestra.run.smithi084.stdout: "state": "up:active", 2024-04-28T12:48:43.088 INFO:teuthology.orchestra.run.smithi084.stdout: "fs_name": "cephfs", 2024-04-28T12:48:43.088 INFO:teuthology.orchestra.run.smithi084.stdout: "rank_uptime": 7.9087817229999997, 2024-04-28T12:48:43.089 INFO:teuthology.orchestra.run.smithi084.stdout: "mdsmap_epoch": 17, 2024-04-28T12:48:43.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch": 90, 2024-04-28T12:48:43.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch_barrier": 90, 2024-04-28T12:48:43.089 INFO:teuthology.orchestra.run.smithi084.stdout: "uptime": 14.615466334000001 2024-04-28T12:48:43.089 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-28T12:48:43.089 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0f9000640 1 -- 172.21.15.84:0/3055993542 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe0d80a3c30 msgr2=0x7fe0d80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0f9000640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe0d80a3c30 0x7fe0d80a60f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4000c00 tx=0x7fe0e40314a0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0f9000640 1 -- 172.21.15.84:0/3055993542 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0f4078530 msgr2=0x7fe0f419f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:43.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0f9000640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0f4078530 0x7fe0f419f6c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00e9f0 tx=0x7fe0dc00eeb0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0f9000640 1 -- 172.21.15.84:0/3055993542 shutdown_connections 2024-04-28T12:48:43.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0f9000640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe0d80a3c30 0x7fe0d80a60f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0f9000640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f4078ed0 0x7fe0f41a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0f9000640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0f4078530 0x7fe0f419f6c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0f9000640 1 --2- 172.21.15.84:0/3055993542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0f40772f0 0x7fe0f419f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:43.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0f9000640 1 -- 172.21.15.84:0/3055993542 >> 172.21.15.84:0/3055993542 conn(0x7fe0f41000f0 msgr2=0x7fe0f4103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:43.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.088+0000 7fe0f9000640 1 -- 172.21.15.84:0/3055993542 shutdown_connections 2024-04-28T12:48:43.091 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:43.089+0000 7fe0f9000640 1 -- 172.21.15.84:0/3055993542 wait complete. 2024-04-28T12:48:43.101 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-28T12:48:43.101 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-28T12:48:43.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-28T12:48:43.531 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:48:43.873 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:43 smithi084 ceph-mon[28081]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-28T12:48:43.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:43 smithi084 ceph-mon[28081]: Cluster is now healthy 2024-04-28T12:48:43.874 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:43 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2119986788' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:43.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:43 smithi179 ceph-mon[63029]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-28T12:48:43.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:43 smithi179 ceph-mon[63029]: Cluster is now healthy 2024-04-28T12:48:43.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:43 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2119986788' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:43.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:43 smithi161 ceph-mon[62387]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-28T12:48:43.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:43 smithi161 ceph-mon[62387]: Cluster is now healthy 2024-04-28T12:48:43.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:43 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2119986788' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:44.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:44 smithi179 ceph-mon[63029]: pgmap v327: 129 pgs: 1 creating+activating, 2 creating+peering, 126 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-04-28T12:48:44.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:44 smithi084 ceph-mon[28081]: pgmap v327: 129 pgs: 1 creating+activating, 2 creating+peering, 126 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-04-28T12:48:44.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:44 smithi161 ceph-mon[62387]: pgmap v327: 129 pgs: 1 creating+activating, 2 creating+peering, 126 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-04-28T12:48:45.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.984+0000 7ff7a5000700 1 -- 172.21.15.84:0/4193175741 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7a0106000 msgr2=0x7ff7a0112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:45.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.984+0000 7ff7a5000700 1 --2- 172.21.15.84:0/4193175741 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7a0106000 0x7ff7a0112510 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff79800b0a0 tx=0x7ff79800b3b0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.984+0000 7ff7a5000700 1 -- 172.21.15.84:0/4193175741 shutdown_connections 2024-04-28T12:48:45.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.984+0000 7ff7a5000700 1 --2- 172.21.15.84:0/4193175741 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7a010b640 0x7ff7a010ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.984+0000 7ff7a5000700 1 --2- 172.21.15.84:0/4193175741 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7a0106000 0x7ff7a0112510 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.984+0000 7ff7a5000700 1 --2- 172.21.15.84:0/4193175741 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7a0105640 0x7ff7a0105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.984+0000 7ff7a5000700 1 -- 172.21.15.84:0/4193175741 >> 172.21.15.84:0/4193175741 conn(0x7ff7a0101130 msgr2=0x7ff7a0103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:45.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.984+0000 7ff7a5000700 1 -- 172.21.15.84:0/4193175741 shutdown_connections 2024-04-28T12:48:45.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.984+0000 7ff7a5000700 1 -- 172.21.15.84:0/4193175741 wait complete. 2024-04-28T12:48:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.985+0000 7ff7a5000700 1 Processor -- start 2024-04-28T12:48:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.985+0000 7ff7a5000700 1 -- start start 2024-04-28T12:48:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.986+0000 7ff7a5000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7a0105640 0x7ff7a01a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.986+0000 7ff7a5000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7a0106000 0x7ff7a01a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.986+0000 7ff7a5000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7a010b640 0x7ff7a019bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.986+0000 7ff7a5000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a0115890 con 0x7ff7a010b640 2024-04-28T12:48:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.986+0000 7ff7a5000700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a0115710 con 0x7ff7a0105640 2024-04-28T12:48:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.986+0000 7ff7a5000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a0115a10 con 0x7ff7a0106000 2024-04-28T12:48:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.986+0000 7ff79e800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7a010b640 0x7ff7a019bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.986+0000 7ff79e800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7a010b640 0x7ff7a019bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52904/0 (socket says 172.21.15.84:52904) 2024-04-28T12:48:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.986+0000 7ff79e800700 1 -- 172.21.15.84:0/1045918285 learned_addr learned my addr 172.21.15.84:0/1045918285 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.986+0000 7ff79d400700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7a0106000 0x7ff7a01a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:45.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.986+0000 7ff79de00700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7a0105640 0x7ff7a01a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:45.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.987+0000 7ff79e800700 1 -- 172.21.15.84:0/1045918285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7a0106000 msgr2=0x7ff7a01a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:45.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.987+0000 7ff79e800700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7a0106000 0x7ff7a01a1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.987+0000 7ff79e800700 1 -- 172.21.15.84:0/1045918285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7a0105640 msgr2=0x7ff7a01a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:45.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.987+0000 7ff79e800700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7a0105640 0x7ff7a01a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:45.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.987+0000 7ff79e800700 1 -- 172.21.15.84:0/1045918285 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a019c3f0 con 0x7ff7a010b640 2024-04-28T12:48:45.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.987+0000 7ff79e800700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7a010b640 0x7ff7a019bb10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff79800b070 tx=0x7ff798009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:45.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.987+0000 7ff796a00700 1 -- 172.21.15.84:0/1045918285 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff798021070 con 0x7ff7a010b640 2024-04-28T12:48:45.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.987+0000 7ff7a5000700 1 -- 172.21.15.84:0/1045918285 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7a019c680 con 0x7ff7a010b640 2024-04-28T12:48:45.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.987+0000 7ff796a00700 1 -- 172.21.15.84:0/1045918285 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff798004e40 con 0x7ff7a010b640 2024-04-28T12:48:45.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.987+0000 7ff796a00700 1 -- 172.21.15.84:0/1045918285 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff798004630 con 0x7ff7a010b640 2024-04-28T12:48:45.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.987+0000 7ff7a5000700 1 -- 172.21.15.84:0/1045918285 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7a01a89c0 con 0x7ff7a010b640 2024-04-28T12:48:45.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.989+0000 7ff7a5000700 1 -- 172.21.15.84:0/1045918285 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7a0106d80 con 0x7ff7a010b640 2024-04-28T12:48:45.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.991+0000 7ff796a00700 1 -- 172.21.15.84:0/1045918285 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff798007980 con 0x7ff7a010b640 2024-04-28T12:48:45.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.992+0000 7ff796a00700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7840a3c40 0x7ff7840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:45.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.992+0000 7ff796a00700 1 -- 172.21.15.84:0/1045918285 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff79800f020 con 0x7ff7a010b640 2024-04-28T12:48:45.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.992+0000 7ff79de00700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7840a3c40 0x7ff7840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:45.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.993+0000 7ff79de00700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7840a3c40 0x7ff7840a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff790009680 tx=0x7ff790009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:45.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:45.997+0000 7ff796a00700 1 -- 172.21.15.84:0/1045918285 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff79802a020 con 0x7ff7a010b640 2024-04-28T12:48:46.269 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.272+0000 7ff7a5000700 1 -- 172.21.15.84:0/1045918285 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff7a019cfd0 con 0x7ff7a010b640 2024-04-28T12:48:46.269 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.273+0000 7ff796a00700 1 -- 172.21.15.84:0/1045918285 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7ff7980074e0 con 0x7ff7a010b640 2024-04-28T12:48:46.270 INFO:teuthology.orchestra.run.smithi084.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-28T12:48:46.272 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.276+0000 7ff7a5000700 1 -- 172.21.15.84:0/1045918285 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7840a3c40 msgr2=0x7ff7840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:46.272 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.276+0000 7ff7a5000700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7840a3c40 0x7ff7840a6100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff790009680 tx=0x7ff790009260 comp rx=0 tx=0).stop 2024-04-28T12:48:46.273 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.276+0000 7ff7a5000700 1 -- 172.21.15.84:0/1045918285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7a010b640 msgr2=0x7ff7a019bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:46.273 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.276+0000 7ff7a5000700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7a010b640 0x7ff7a019bb10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff79800b070 tx=0x7ff798009c90 comp rx=0 tx=0).stop 2024-04-28T12:48:46.273 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.276+0000 7ff7a5000700 1 -- 172.21.15.84:0/1045918285 shutdown_connections 2024-04-28T12:48:46.273 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.276+0000 7ff7a5000700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7840a3c40 0x7ff7840a6100 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.273 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.276+0000 7ff7a5000700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7a0105640 0x7ff7a01a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.273 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.276+0000 7ff7a5000700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7a010b640 0x7ff7a019bb10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.273 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.276+0000 7ff7a5000700 1 --2- 172.21.15.84:0/1045918285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7a0106000 0x7ff7a01a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:46.273 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.276+0000 7ff7a5000700 1 -- 172.21.15.84:0/1045918285 >> 172.21.15.84:0/1045918285 conn(0x7ff7a0101130 msgr2=0x7ff7a01017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:46.273 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.276+0000 7ff7a5000700 1 -- 172.21.15.84:0/1045918285 shutdown_connections 2024-04-28T12:48:46.273 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:46.276+0000 7ff7a5000700 1 -- 172.21.15.84:0/1045918285 wait complete. 2024-04-28T12:48:46.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:46 smithi179 ceph-mon[63029]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-04-28T12:48:46.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:46 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1045918285' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T12:48:46.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:46 smithi084 ceph-mon[28081]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-04-28T12:48:46.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:46 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1045918285' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T12:48:46.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:46 smithi161 ceph-mon[62387]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-04-28T12:48:46.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:46 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1045918285' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T12:48:47.346 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-28T12:48:47.346 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-28T12:48:47.346 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi084.front.sepia.ceph.com 2024-04-28T12:48:47.346 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-28T12:48:47.346 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-28T12:48:47.346 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-28T12:48:47.346 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-28T12:48:47.346 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-28T12:48:47.346 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-28T12:48:47.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:48:47.347 DEBUG:teuthology.orchestra.run.smithi084:> ip addr 2024-04-28T12:48:47.403 INFO:teuthology.orchestra.run.smithi084.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-28T12:48:47.404 INFO:teuthology.orchestra.run.smithi084.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-28T12:48:47.404 INFO:teuthology.orchestra.run.smithi084.stdout: inet 127.0.0.1/8 scope host lo 2024-04-28T12:48:47.404 INFO:teuthology.orchestra.run.smithi084.stdout: valid_lft forever preferred_lft forever 2024-04-28T12:48:47.404 INFO:teuthology.orchestra.run.smithi084.stdout: inet6 ::1/128 scope host 2024-04-28T12:48:47.404 INFO:teuthology.orchestra.run.smithi084.stdout: valid_lft forever preferred_lft forever 2024-04-28T12:48:47.404 INFO:teuthology.orchestra.run.smithi084.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T12:48:47.404 INFO:teuthology.orchestra.run.smithi084.stdout: link/ether 0c:c4:7a:d8:da:68 brd ff:ff:ff:ff:ff:ff 2024-04-28T12:48:47.404 INFO:teuthology.orchestra.run.smithi084.stdout: altname enp7s0f0 2024-04-28T12:48:47.404 INFO:teuthology.orchestra.run.smithi084.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T12:48:47.404 INFO:teuthology.orchestra.run.smithi084.stdout: link/ether 0c:c4:7a:d8:da:69 brd ff:ff:ff:ff:ff:ff 2024-04-28T12:48:47.404 INFO:teuthology.orchestra.run.smithi084.stdout: altname enp7s0f1 2024-04-28T12:48:47.404 INFO:teuthology.orchestra.run.smithi084.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T12:48:47.405 INFO:teuthology.orchestra.run.smithi084.stdout: link/ether 0c:c4:7a:8f:d7:50 brd ff:ff:ff:ff:ff:ff 2024-04-28T12:48:47.405 INFO:teuthology.orchestra.run.smithi084.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-28T12:48:47.405 INFO:teuthology.orchestra.run.smithi084.stdout: link/ether 0c:c4:7a:8f:d7:51 brd ff:ff:ff:ff:ff:ff 2024-04-28T12:48:47.405 INFO:teuthology.orchestra.run.smithi084.stdout: inet 172.21.15.84/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-28T12:48:47.405 INFO:teuthology.orchestra.run.smithi084.stdout: valid_lft 41842sec preferred_lft 41842sec 2024-04-28T12:48:47.405 INFO:teuthology.orchestra.run.smithi084.stdout: inet6 fe80::ec4:7aff:fe8f:d751/64 scope link 2024-04-28T12:48:47.405 INFO:teuthology.orchestra.run.smithi084.stdout: valid_lft forever preferred_lft forever 2024-04-28T12:48:47.406 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-28T12:48:47.406 DEBUG:teuthology.orchestra.run.smithi084:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:48:47.406 DEBUG:teuthology.orchestra.run.smithi084:> set -e 2024-04-28T12:48:47.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip link add name ceph-brx type bridge 2024-04-28T12:48:47.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip addr flush dev ceph-brx 2024-04-28T12:48:47.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip link set ceph-brx up 2024-04-28T12:48:47.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-28T12:48:47.407 DEBUG:teuthology.orchestra.run.smithi084:> ') 2024-04-28T12:48:47.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:48:47.868 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:48:47.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:48:47.878 DEBUG:teuthology.orchestra.run.smithi084:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-28T12:48:47.949 INFO:teuthology.orchestra.run.smithi084.stdout:1 2024-04-28T12:48:47.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:48:47.953 DEBUG:teuthology.orchestra.run.smithi084:> route 2024-04-28T12:48:48.003 INFO:teuthology.orchestra.run.smithi084.stdout:Kernel IP routing table 2024-04-28T12:48:48.004 INFO:teuthology.orchestra.run.smithi084.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-28T12:48:48.004 INFO:teuthology.orchestra.run.smithi084.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-28T12:48:48.004 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-28T12:48:48.004 INFO:teuthology.orchestra.run.smithi084.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-28T12:48:48.005 DEBUG:teuthology.orchestra.run.smithi084:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:48:48.005 DEBUG:teuthology.orchestra.run.smithi084:> set -e 2024-04-28T12:48:48.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-28T12:48:48.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-28T12:48:48.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-28T12:48:48.005 DEBUG:teuthology.orchestra.run.smithi084:> ') 2024-04-28T12:48:48.097 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:48:48.666 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:48 smithi084 ceph-mon[28081]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-04-28T12:48:48.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:48:48.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:48:48.705 DEBUG:teuthology.orchestra.run.smithi084:> ip netns list 2024-04-28T12:48:48.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:48:48.758 DEBUG:teuthology.orchestra.run.smithi084:> ip netns list-id 2024-04-28T12:48:48.808 DEBUG:teuthology.orchestra.run.smithi084:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:48:48.808 DEBUG:teuthology.orchestra.run.smithi084:> set -e 2024-04-28T12:48:48.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T12:48:48.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-28T12:48:48.808 DEBUG:teuthology.orchestra.run.smithi084:> ') 2024-04-28T12:48:48.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:48:48.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:48 smithi179 ceph-mon[63029]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-04-28T12:48:48.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:48:48.991 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-28T12:48:48.992 DEBUG:teuthology.orchestra.run.smithi084:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:48:48.992 DEBUG:teuthology.orchestra.run.smithi084:> set -e 2024-04-28T12:48:48.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-28T12:48:48.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-28T12:48:48.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-28T12:48:48.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-28T12:48:48.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-28T12:48:48.992 DEBUG:teuthology.orchestra.run.smithi084:> ') 2024-04-28T12:48:48.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:48 smithi161 ceph-mon[62387]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-04-28T12:48:49.088 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:48:49.276 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:48:49.286 DEBUG:teuthology.orchestra.run.smithi084:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:48:49.286 DEBUG:teuthology.orchestra.run.smithi084:> set -e 2024-04-28T12:48:49.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip link set brx.0 up 2024-04-28T12:48:49.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip link set dev brx.0 master ceph-brx 2024-04-28T12:48:49.287 DEBUG:teuthology.orchestra.run.smithi084:> ') 2024-04-28T12:48:49.383 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:48:49.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:48:49.491 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-28T12:48:49.492 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-28T12:48:49.492 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-28T12:48:49.540 INFO:teuthology.orchestra.run.smithi084.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-28T12:48:49.541 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-28T12:48:49.541 DEBUG:teuthology.orchestra.run.smithi084:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-28T12:48:49.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo modprobe fuse 2024-04-28T12:48:49.713 DEBUG:teuthology.orchestra.run.smithi084:> cat /proc/self/mounts | awk '{print $2}' 2024-04-28T12:48:49.763 INFO:teuthology.orchestra.run.smithi084.stdout:/proc 2024-04-28T12:48:49.763 INFO:teuthology.orchestra.run.smithi084.stdout:/sys 2024-04-28T12:48:49.763 INFO:teuthology.orchestra.run.smithi084.stdout:/dev 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/kernel/security 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/shm 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/pts 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/run 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/cgroup 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/pstore 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/bpf 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/ 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/selinux 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/proc/sys/fs/binfmt_misc 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/hugepages 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/mqueue 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/kernel/debug 2024-04-28T12:48:49.764 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/kernel/tracing 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/fuse/connections 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/kernel/config 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/run/credentials/systemd-sysctl.service 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/nfs/rpc_pipefs 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/run/user/1000 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/ceph 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/14066d0c1e9a4c46cf420988ca62bd379eea21b17077e208d6ab60f9174e6dfb/merged 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/e9e674be30c93af2927a9f045bead0315ddad4ba819dee5491a2dc7bf55cde88/merged 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/run/user/0 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/proc/sys/fs/binfmt_misc 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/7e1e1d14463382ae011e9c37e2c695d6a8f91698be105bc45ee45bba10fc9fbd/merged 2024-04-28T12:48:49.765 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/988ae59056e61be0ce719abf4d884ea9b1cfffbfb3bc7f9c26a5037f763c81c2/merged 2024-04-28T12:48:49.766 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/94bd54bbfb729251af24c3392c0d43950830a8177fd59e1ccb7ebbff00d2d13c/merged 2024-04-28T12:48:49.766 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/add277074797939697c36697316d4a23309a659872aa955401b31cbb2a919136/merged 2024-04-28T12:48:49.766 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/82b1d0097c29cbc4ec971ed2a2804060f98735922da6f3fbfb0f5225ea3ee869/merged 2024-04-28T12:48:49.766 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/b596e6f5b919c1b5e046db407ee954d4df5733f5ece2455b8db4b969e15fc19f/merged 2024-04-28T12:48:49.766 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/b60e741e39ae96466ad43d8b9e1cf3811a57d5d87dccea52326388e45d931a73/merged 2024-04-28T12:48:49.766 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/bae603311ee9b9290aceef5ffc90c14332a04a5c0398aae9f12a0d42750ea3f6/merged 2024-04-28T12:48:49.766 INFO:teuthology.orchestra.run.smithi084.stdout:/run/netns 2024-04-28T12:48:49.766 INFO:teuthology.orchestra.run.smithi084.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T12:48:49.766 INFO:teuthology.orchestra.run.smithi084.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T12:48:49.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:48:49.767 DEBUG:teuthology.orchestra.run.smithi084:> ls /sys/fs/fuse/connections 2024-04-28T12:48:49.814 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-28T12:48:49.815 DEBUG:teuthology.orchestra.run.smithi084:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-28T12:48:49.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo modprobe fuse 2024-04-28T12:48:49.938 DEBUG:teuthology.orchestra.run.smithi084:> cat /proc/self/mounts | awk '{print $2}' 2024-04-28T12:48:49.982 INFO:teuthology.orchestra.run.smithi084.stdout:/proc 2024-04-28T12:48:49.982 INFO:teuthology.orchestra.run.smithi084.stdout:/sys 2024-04-28T12:48:49.982 INFO:teuthology.orchestra.run.smithi084.stdout:/dev 2024-04-28T12:48:49.982 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/kernel/security 2024-04-28T12:48:49.982 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/shm 2024-04-28T12:48:49.982 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/pts 2024-04-28T12:48:49.982 INFO:teuthology.orchestra.run.smithi084.stdout:/run 2024-04-28T12:48:49.982 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/cgroup 2024-04-28T12:48:49.982 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/pstore 2024-04-28T12:48:49.982 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/bpf 2024-04-28T12:48:49.982 INFO:teuthology.orchestra.run.smithi084.stdout:/ 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/selinux 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/proc/sys/fs/binfmt_misc 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/hugepages 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/mqueue 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/kernel/debug 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/kernel/tracing 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/fuse/connections 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/kernel/config 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/run/credentials/systemd-sysctl.service 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/nfs/rpc_pipefs 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/run/user/1000 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/ceph 2024-04-28T12:48:49.983 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/14066d0c1e9a4c46cf420988ca62bd379eea21b17077e208d6ab60f9174e6dfb/merged 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/e9e674be30c93af2927a9f045bead0315ddad4ba819dee5491a2dc7bf55cde88/merged 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/run/user/0 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/proc/sys/fs/binfmt_misc 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/7e1e1d14463382ae011e9c37e2c695d6a8f91698be105bc45ee45bba10fc9fbd/merged 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/988ae59056e61be0ce719abf4d884ea9b1cfffbfb3bc7f9c26a5037f763c81c2/merged 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/94bd54bbfb729251af24c3392c0d43950830a8177fd59e1ccb7ebbff00d2d13c/merged 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/add277074797939697c36697316d4a23309a659872aa955401b31cbb2a919136/merged 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/82b1d0097c29cbc4ec971ed2a2804060f98735922da6f3fbfb0f5225ea3ee869/merged 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/b596e6f5b919c1b5e046db407ee954d4df5733f5ece2455b8db4b969e15fc19f/merged 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/b60e741e39ae96466ad43d8b9e1cf3811a57d5d87dccea52326388e45d931a73/merged 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/bae603311ee9b9290aceef5ffc90c14332a04a5c0398aae9f12a0d42750ea3f6/merged 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/run/netns 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T12:48:49.984 INFO:teuthology.orchestra.run.smithi084.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T12:48:49.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:48:49.985 DEBUG:teuthology.orchestra.run.smithi084:> ls /sys/fs/fuse/connections 2024-04-28T12:48:50.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi084.stderr:2024-04-28T12:48:50.081+0000 7f9791198480 -1 init, newargv = 0x56313e18b4f0 newargc=15 2024-04-28T12:48:50.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi084.stderr:ceph-fuse[68641]: starting ceph client 2024-04-28T12:48:50.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi084.stderr:ceph-fuse[68641]: starting fuse 2024-04-28T12:48:50.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:50 smithi179 ceph-mon[63029]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-28T12:48:50.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:50 smithi084 ceph-mon[28081]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-28T12:48:50.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:50 smithi161 ceph-mon[62387]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-28T12:48:51.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo modprobe fuse 2024-04-28T12:48:51.263 DEBUG:teuthology.orchestra.run.smithi084:> cat /proc/self/mounts | awk '{print $2}' 2024-04-28T12:48:51.322 INFO:teuthology.orchestra.run.smithi084.stdout:/proc 2024-04-28T12:48:51.322 INFO:teuthology.orchestra.run.smithi084.stdout:/sys 2024-04-28T12:48:51.322 INFO:teuthology.orchestra.run.smithi084.stdout:/dev 2024-04-28T12:48:51.322 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/kernel/security 2024-04-28T12:48:51.322 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/shm 2024-04-28T12:48:51.322 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/pts 2024-04-28T12:48:51.322 INFO:teuthology.orchestra.run.smithi084.stdout:/run 2024-04-28T12:48:51.322 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/cgroup 2024-04-28T12:48:51.322 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/pstore 2024-04-28T12:48:51.322 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/bpf 2024-04-28T12:48:51.322 INFO:teuthology.orchestra.run.smithi084.stdout:/ 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/selinux 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/proc/sys/fs/binfmt_misc 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/hugepages 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/mqueue 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/kernel/debug 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/kernel/tracing 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/fs/fuse/connections 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/sys/kernel/config 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/run/credentials/systemd-sysctl.service 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/nfs/rpc_pipefs 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/run/user/1000 2024-04-28T12:48:51.323 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/ceph 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/14066d0c1e9a4c46cf420988ca62bd379eea21b17077e208d6ab60f9174e6dfb/merged 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/e9e674be30c93af2927a9f045bead0315ddad4ba819dee5491a2dc7bf55cde88/merged 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/run/user/0 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/proc/sys/fs/binfmt_misc 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/7e1e1d14463382ae011e9c37e2c695d6a8f91698be105bc45ee45bba10fc9fbd/merged 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/988ae59056e61be0ce719abf4d884ea9b1cfffbfb3bc7f9c26a5037f763c81c2/merged 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/94bd54bbfb729251af24c3392c0d43950830a8177fd59e1ccb7ebbff00d2d13c/merged 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/add277074797939697c36697316d4a23309a659872aa955401b31cbb2a919136/merged 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/82b1d0097c29cbc4ec971ed2a2804060f98735922da6f3fbfb0f5225ea3ee869/merged 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/b596e6f5b919c1b5e046db407ee954d4df5733f5ece2455b8db4b969e15fc19f/merged 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/b60e741e39ae96466ad43d8b9e1cf3811a57d5d87dccea52326388e45d931a73/merged 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/var/lib/containers/storage/overlay/bae603311ee9b9290aceef5ffc90c14332a04a5c0398aae9f12a0d42750ea3f6/merged 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/run/netns 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T12:48:51.324 INFO:teuthology.orchestra.run.smithi084.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-28T12:48:51.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:48:51.325 DEBUG:teuthology.orchestra.run.smithi084:> ls /sys/fs/fuse/connections 2024-04-28T12:48:51.376 INFO:teuthology.orchestra.run.smithi084.stdout:114 2024-04-28T12:48:51.376 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-04-28T12:48:51.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo stdin-killer -- python3 -c ' 2024-04-28T12:48:51.376 DEBUG:teuthology.orchestra.run.smithi084:> import glob 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> import re 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> import os 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> import subprocess 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> def _find_admin_socket(client_name): 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> files = glob.glob(asok_path) 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> # Given a non-glob path, it better be there 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> if "*" not in asok_path: 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> assert(len(files) == 1) 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> return files[0] 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> for f in files: 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-28T12:48:51.377 DEBUG:teuthology.orchestra.run.smithi084:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-28T12:48:51.378 DEBUG:teuthology.orchestra.run.smithi084:> contents = proc_f.read() 2024-04-28T12:48:51.378 DEBUG:teuthology.orchestra.run.smithi084:> if mountpoint in contents: 2024-04-28T12:48:51.378 DEBUG:teuthology.orchestra.run.smithi084:> return f 2024-04-28T12:48:51.378 DEBUG:teuthology.orchestra.run.smithi084:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-28T12:48:51.378 DEBUG:teuthology.orchestra.run.smithi084:> 2024-04-28T12:48:51.378 DEBUG:teuthology.orchestra.run.smithi084:> print(_find_admin_socket("client.admin")) 2024-04-28T12:48:51.378 DEBUG:teuthology.orchestra.run.smithi084:> ' 2024-04-28T12:48:51.538 INFO:teuthology.orchestra.run.smithi084.stdout:/var/run/ceph/ceph-client.admin.68641.asok 2024-04-28T12:48:51.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:48:51.555 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.68641.asok 2024-04-28T12:48:51.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:48:51.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.68641.asok status 2024-04-28T12:48:51.781 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "metadata": { 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "ceph_sha1": "bc612ffb9c1d5bc27536bc917a705262a36d4387", 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "entity_id": "admin", 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "hostname": "smithi084", 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "pid": "68641", 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "root": "/" 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "dentry_count": 0, 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "dentry_pinned_count": 0, 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "id": 14601, 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "inst": { 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "name": { 2024-04-28T12:48:51.782 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "client", 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: "num": 14601 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": { 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "192.168.0.1:0", 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3671693528 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": { 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v1", 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "192.168.0.1:0", 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3671693528 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: "inst_str": "client.14601 192.168.0.1:0/3671693528", 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: "addr_str": "192.168.0.1:0/3671693528", 2024-04-28T12:48:51.783 INFO:teuthology.orchestra.run.smithi084.stdout: "inode_count": 1, 2024-04-28T12:48:51.784 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_epoch": 17, 2024-04-28T12:48:51.784 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_epoch": 90, 2024-04-28T12:48:51.784 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_epoch_barrier": 0, 2024-04-28T12:48:51.784 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklisted": false, 2024-04-28T12:48:51.784 INFO:teuthology.orchestra.run.smithi084.stdout: "fs_name": "cephfs" 2024-04-28T12:48:51.784 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-28T12:48:51.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:48:51.796 DEBUG:teuthology.orchestra.run.smithi084:> stat --file-system '--printf=%T 2024-04-28T12:48:51.796 DEBUG:teuthology.orchestra.run.smithi084:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-28T12:48:51.851 INFO:teuthology.orchestra.run.smithi084.stdout:fuseblk 2024-04-28T12:48:51.851 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-28T12:48:51.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:48:51.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-28T12:48:51.938 DEBUG:teuthology.orchestra.run.smithi084:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-28T12:48:52.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-28T12:48:52.084 INFO:teuthology.orchestra.run.smithi084.stdout:# file: . 2024-04-28T12:48:52.084 INFO:teuthology.orchestra.run.smithi084.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-28T12:48:52.084 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:52.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:48:52.095 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:48:52.095 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T12:48:52.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:48:52.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.361+0000 7f9e56000640 1 -- 172.21.15.84:0/4227708684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e50075330 msgr2=0x7f9e50075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.361+0000 7f9e56000640 1 --2- 172.21.15.84:0/4227708684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e50075330 0x7f9e50075790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c009900 tx=0x7f9e3c02a480 comp rx=0 tx=0).stop 2024-04-28T12:48:52.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.361+0000 7f9e56000640 1 -- 172.21.15.84:0/4227708684 shutdown_connections 2024-04-28T12:48:52.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.361+0000 7f9e56000640 1 --2- 172.21.15.84:0/4227708684 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e5010a5d0 0x7f9e5010ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.361+0000 7f9e56000640 1 --2- 172.21.15.84:0/4227708684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e50075330 0x7f9e50075790 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.361+0000 7f9e56000640 1 --2- 172.21.15.84:0/4227708684 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e50076910 0x7f9e50074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.361+0000 7f9e56000640 1 -- 172.21.15.84:0/4227708684 >> 172.21.15.84:0/4227708684 conn(0x7f9e500fdcb0 msgr2=0x7f9e501000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:52.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.361+0000 7f9e56000640 1 -- 172.21.15.84:0/4227708684 shutdown_connections 2024-04-28T12:48:52.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.361+0000 7f9e56000640 1 -- 172.21.15.84:0/4227708684 wait complete. 2024-04-28T12:48:52.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.362+0000 7f9e56000640 1 Processor -- start 2024-04-28T12:48:52.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.362+0000 7f9e56000640 1 -- start start 2024-04-28T12:48:52.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.362+0000 7f9e56000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e50075330 0x7f9e5019ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.362+0000 7f9e56000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e50076910 0x7f9e5019b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.362+0000 7f9e56000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e5010a5d0 0x7f9e501a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.362+0000 7f9e56000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501104a0 con 0x7f9e50076910 2024-04-28T12:48:52.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.362+0000 7f9e56000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9e50110320 con 0x7f9e50075330 2024-04-28T12:48:52.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.362+0000 7f9e56000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f9e50110620 con 0x7f9e5010a5d0 2024-04-28T12:48:52.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.363+0000 7f9e54c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e50075330 0x7f9e5019ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.363+0000 7f9e54c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e50075330 0x7f9e5019ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:55794/0 (socket says 172.21.15.84:55794) 2024-04-28T12:48:52.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.363+0000 7f9e54c00640 1 -- 172.21.15.84:0/115414263 learned_addr learned my addr 172.21.15.84:0/115414263 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:52.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.363+0000 7f9e4fe00640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e50076910 0x7f9e5019b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.363+0000 7f9e55600640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e5010a5d0 0x7f9e501a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.363+0000 7f9e54c00640 1 -- 172.21.15.84:0/115414263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e5010a5d0 msgr2=0x7f9e501a2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.363+0000 7f9e54c00640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e5010a5d0 0x7f9e501a2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.363+0000 7f9e54c00640 1 -- 172.21.15.84:0/115414263 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e50076910 msgr2=0x7f9e5019b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.363+0000 7f9e54c00640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e50076910 0x7f9e5019b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.363+0000 7f9e54c00640 1 -- 172.21.15.84:0/115414263 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e501a2790 con 0x7f9e50075330 2024-04-28T12:48:52.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.364+0000 7f9e54c00640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e50075330 0x7f9e5019ab70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400e9f0 tx=0x7f9e4400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:52.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.364+0000 7f9e4d600640 1 -- 172.21.15.84:0/115414263 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4400cd80 con 0x7f9e50075330 2024-04-28T12:48:52.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.364+0000 7f9e56000640 1 -- 172.21.15.84:0/115414263 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e501a2a80 con 0x7f9e50075330 2024-04-28T12:48:52.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.364+0000 7f9e4d600640 1 -- 172.21.15.84:0/115414263 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e440108e0 con 0x7f9e50075330 2024-04-28T12:48:52.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.364+0000 7f9e4d600640 1 -- 172.21.15.84:0/115414263 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e44018980 con 0x7f9e50075330 2024-04-28T12:48:52.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.364+0000 7f9e56000640 1 -- 172.21.15.84:0/115414263 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e501a2ff0 con 0x7f9e50075330 2024-04-28T12:48:52.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.365+0000 7f9e56000640 1 -- 172.21.15.84:0/115414263 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e50109390 con 0x7f9e50075330 2024-04-28T12:48:52.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.367+0000 7f9e4d600640 1 -- 172.21.15.84:0/115414263 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9e44010430 con 0x7f9e50075330 2024-04-28T12:48:52.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.369+0000 7f9e4d600640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f9e380a3bb0 0x7f9e380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.369+0000 7f9e4d600640 1 -- 172.21.15.84:0/115414263 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f9e440d0e20 con 0x7f9e50075330 2024-04-28T12:48:52.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.369+0000 7f9e4fe00640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f9e380a3bb0 0x7f9e380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.370+0000 7f9e4fe00640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f9e380a3bb0 0x7f9e380a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c031500 tx=0x7f9e3c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:52.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.371+0000 7f9e4d600640 1 -- 172.21.15.84:0/115414263 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9e4409cf10 con 0x7f9e50075330 2024-04-28T12:48:52.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.573+0000 7f9e56000640 1 -- 172.21.15.84:0/115414263 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e5002dd30 con 0x7f9e50075330 2024-04-28T12:48:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.575+0000 7f9e4d600640 1 -- 172.21.15.84:0/115414263 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24117 (secure 0 0 0) 0x7f9e440a1210 con 0x7f9e50075330 2024-04-28T12:48:52.572 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:52.573 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":90,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:48:40.059255+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:48:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.578+0000 7f9e56000640 1 -- 172.21.15.84:0/115414263 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f9e380a3bb0 msgr2=0x7f9e380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.578+0000 7f9e56000640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f9e380a3bb0 0x7f9e380a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c031500 tx=0x7f9e3c005a20 comp rx=0 tx=0).stop 2024-04-28T12:48:52.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.578+0000 7f9e56000640 1 -- 172.21.15.84:0/115414263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e50075330 msgr2=0x7f9e5019ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.578+0000 7f9e56000640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e50075330 0x7f9e5019ab70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400e9f0 tx=0x7f9e4400eeb0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.578+0000 7f9e56000640 1 -- 172.21.15.84:0/115414263 shutdown_connections 2024-04-28T12:48:52.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.578+0000 7f9e56000640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f9e380a3bb0 0x7f9e380a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.578+0000 7f9e56000640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e5010a5d0 0x7f9e501a2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.578+0000 7f9e56000640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e50076910 0x7f9e5019b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.578+0000 7f9e56000640 1 --2- 172.21.15.84:0/115414263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e50075330 0x7f9e5019ab70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.579+0000 7f9e56000640 1 -- 172.21.15.84:0/115414263 >> 172.21.15.84:0/115414263 conn(0x7f9e500fdcb0 msgr2=0x7f9e50106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.579+0000 7f9e56000640 1 -- 172.21.15.84:0/115414263 shutdown_connections 2024-04-28T12:48:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.579+0000 7f9e56000640 1 -- 172.21.15.84:0/115414263 wait complete. 2024-04-28T12:48:52.594 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:43:15.608080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T12:48:32.037809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T12:48:33.240166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T12:48:37.567238+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T12:48:52.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:48:52.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.805+0000 7feca6c00640 1 -- 172.21.15.84:0/389564430 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feca0109fc0 msgr2=0x7feca010a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.805+0000 7feca6c00640 1 --2- 172.21.15.84:0/389564430 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feca0109fc0 0x7feca010a440 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fec8c009900 tx=0x7fec8c02a3f0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.805+0000 7feca6c00640 1 -- 172.21.15.84:0/389564430 shutdown_connections 2024-04-28T12:48:52.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.805+0000 7feca6c00640 1 --2- 172.21.15.84:0/389564430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca010a980 0x7feca01112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.805+0000 7feca6c00640 1 --2- 172.21.15.84:0/389564430 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feca0109fc0 0x7feca010a440 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.805+0000 7feca6c00640 1 --2- 172.21.15.84:0/389564430 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feca0108e70 0x7feca0109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.805+0000 7feca6c00640 1 -- 172.21.15.84:0/389564430 >> 172.21.15.84:0/389564430 conn(0x7feca0077e00 msgr2=0x7feca007a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:52.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.805+0000 7feca6c00640 1 -- 172.21.15.84:0/389564430 shutdown_connections 2024-04-28T12:48:52.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.805+0000 7feca6c00640 1 -- 172.21.15.84:0/389564430 wait complete. 2024-04-28T12:48:52.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.806+0000 7feca6c00640 1 Processor -- start 2024-04-28T12:48:52.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.806+0000 7feca6c00640 1 -- start start 2024-04-28T12:48:52.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.807+0000 7feca6c00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca0108e70 0x7feca019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.807+0000 7feca6c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feca0109fc0 0x7feca019f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.807+0000 7feca6c00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feca010a980 0x7feca01a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.807+0000 7feca6c00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7feca01149f0 con 0x7feca010a980 2024-04-28T12:48:52.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.807+0000 7feca6c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feca0114870 con 0x7feca0109fc0 2024-04-28T12:48:52.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.807+0000 7feca6c00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7feca0114b70 con 0x7feca0108e70 2024-04-28T12:48:52.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.807+0000 7feca6200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feca010a980 0x7feca01a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.807+0000 7feca4e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feca0109fc0 0x7feca019f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.807+0000 7feca5800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca0108e70 0x7feca019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.807+0000 7feca4e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feca0109fc0 0x7feca019f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:55812/0 (socket says 172.21.15.84:55812) 2024-04-28T12:48:52.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.807+0000 7feca4e00640 1 -- 172.21.15.84:0/192889666 learned_addr learned my addr 172.21.15.84:0/192889666 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:52.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.808+0000 7feca5800640 1 -- 172.21.15.84:0/192889666 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feca0109fc0 msgr2=0x7feca019f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.808+0000 7feca5800640 1 --2- 172.21.15.84:0/192889666 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feca0109fc0 0x7feca019f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.808+0000 7feca5800640 1 -- 172.21.15.84:0/192889666 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feca010a980 msgr2=0x7feca01a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:52.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.808+0000 7feca5800640 1 --2- 172.21.15.84:0/192889666 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feca010a980 0x7feca01a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:52.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.808+0000 7feca5800640 1 -- 172.21.15.84:0/192889666 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca01a6c80 con 0x7feca0108e70 2024-04-28T12:48:52.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.808+0000 7feca5800640 1 --2- 172.21.15.84:0/192889666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca0108e70 0x7feca019f060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fec9400e9f0 tx=0x7fec9400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:52.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.808+0000 7fec9e000640 1 -- 172.21.15.84:0/192889666 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9400ccc0 con 0x7feca0108e70 2024-04-28T12:48:52.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.808+0000 7fec9e000640 1 -- 172.21.15.84:0/192889666 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec940108e0 con 0x7feca0108e70 2024-04-28T12:48:52.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.808+0000 7feca6c00640 1 -- 172.21.15.84:0/192889666 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca01a6f70 con 0x7feca0108e70 2024-04-28T12:48:52.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.808+0000 7fec9e000640 1 -- 172.21.15.84:0/192889666 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec94018880 con 0x7feca0108e70 2024-04-28T12:48:52.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.808+0000 7feca6c00640 1 -- 172.21.15.84:0/192889666 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca01a73d0 con 0x7feca0108e70 2024-04-28T12:48:52.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.810+0000 7feca6c00640 1 -- 172.21.15.84:0/192889666 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feca0110180 con 0x7feca0108e70 2024-04-28T12:48:52.808 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.811+0000 7fec9e000640 1 -- 172.21.15.84:0/192889666 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fec94010430 con 0x7feca0108e70 2024-04-28T12:48:52.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.813+0000 7fec9e000640 1 --2- 172.21.15.84:0/192889666 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fec880a3bb0 0x7fec880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:52.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.813+0000 7fec9e000640 1 -- 172.21.15.84:0/192889666 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fec940d0990 con 0x7feca0108e70 2024-04-28T12:48:52.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.813+0000 7feca4e00640 1 --2- 172.21.15.84:0/192889666 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fec880a3bb0 0x7fec880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:52.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.814+0000 7feca4e00640 1 --2- 172.21.15.84:0/192889666 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fec880a3bb0 0x7fec880a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fec8c000c00 tx=0x7fec8c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:52.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:52.817+0000 7fec9e000640 1 -- 172.21.15.84:0/192889666 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fec9409ca80 con 0x7feca0108e70 2024-04-28T12:48:52.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:52 smithi179 ceph-mon[63029]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:52.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:52 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/115414263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:52.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:52 smithi084 ceph-mon[28081]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:52.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:52 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/115414263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:52.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:52 smithi161 ceph-mon[62387]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:52.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:52 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/115414263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:53.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.017+0000 7feca6c00640 1 -- 172.21.15.84:0/192889666 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7feca0063560 con 0x7feca0108e70 2024-04-28T12:48:53.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.018+0000 7fec9e000640 1 -- 172.21.15.84:0/192889666 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fec940a0d80 con 0x7feca0108e70 2024-04-28T12:48:53.015 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-04-28T12:48:53.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.021+0000 7feca6c00640 1 -- 172.21.15.84:0/192889666 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fec880a3bb0 msgr2=0x7fec880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:53.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.021+0000 7feca6c00640 1 --2- 172.21.15.84:0/192889666 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fec880a3bb0 0x7fec880a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fec8c000c00 tx=0x7fec8c005a20 comp rx=0 tx=0).stop 2024-04-28T12:48:53.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.021+0000 7feca6c00640 1 -- 172.21.15.84:0/192889666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca0108e70 msgr2=0x7feca019f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:53.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.021+0000 7feca6c00640 1 --2- 172.21.15.84:0/192889666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca0108e70 0x7feca019f060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fec9400e9f0 tx=0x7fec9400eeb0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.022+0000 7feca6c00640 1 -- 172.21.15.84:0/192889666 shutdown_connections 2024-04-28T12:48:53.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.022+0000 7feca6c00640 1 --2- 172.21.15.84:0/192889666 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fec880a3bb0 0x7fec880a6070 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.022+0000 7feca6c00640 1 --2- 172.21.15.84:0/192889666 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feca010a980 0x7feca01a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.022+0000 7feca6c00640 1 --2- 172.21.15.84:0/192889666 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feca0109fc0 0x7feca019f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.022+0000 7feca6c00640 1 --2- 172.21.15.84:0/192889666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca0108e70 0x7feca019f060 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.022+0000 7feca6c00640 1 -- 172.21.15.84:0/192889666 >> 172.21.15.84:0/192889666 conn(0x7feca0077e00 msgr2=0x7feca010d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:53.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.022+0000 7feca6c00640 1 -- 172.21.15.84:0/192889666 shutdown_connections 2024-04-28T12:48:53.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.022+0000 7feca6c00640 1 -- 172.21.15.84:0/192889666 wait complete. 2024-04-28T12:48:53.038 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T12:48:53.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.245+0000 7f4f8e600640 1 -- 172.21.15.84:0/3558114317 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f8810ee90 msgr2=0x7f4f88111280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:53.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.245+0000 7f4f8e600640 1 --2- 172.21.15.84:0/3558114317 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f8810ee90 0x7f4f88111280 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4f80009cd0 tx=0x7f4f8002a8f0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.245+0000 7f4f8e600640 1 -- 172.21.15.84:0/3558114317 shutdown_connections 2024-04-28T12:48:53.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.245+0000 7f4f8e600640 1 --2- 172.21.15.84:0/3558114317 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f8810ee90 0x7f4f88111280 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.245+0000 7f4f8e600640 1 --2- 172.21.15.84:0/3558114317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f88075390 0x7f4f88075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.245+0000 7f4f8e600640 1 --2- 172.21.15.84:0/3558114317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f88076970 0x7f4f88074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.245+0000 7f4f8e600640 1 -- 172.21.15.84:0/3558114317 >> 172.21.15.84:0/3558114317 conn(0x7f4f880fde30 msgr2=0x7f4f88100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:53.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.245+0000 7f4f8e600640 1 -- 172.21.15.84:0/3558114317 shutdown_connections 2024-04-28T12:48:53.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.245+0000 7f4f8e600640 1 -- 172.21.15.84:0/3558114317 wait complete. 2024-04-28T12:48:53.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.246+0000 7f4f8e600640 1 Processor -- start 2024-04-28T12:48:53.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.246+0000 7f4f8e600640 1 -- start start 2024-04-28T12:48:53.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8e600640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f88075390 0x7f4f8819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:53.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8e600640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f88076970 0x7f4f8819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:53.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8e600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f8810ee90 0x7f4f881a6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:53.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8e600640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4f881149a0 con 0x7f4f88075390 2024-04-28T12:48:53.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8e600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4f88114820 con 0x7f4f8810ee90 2024-04-28T12:48:53.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8e600640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4f88114b20 con 0x7f4f88076970 2024-04-28T12:48:53.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8d200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f88075390 0x7f4f8819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:53.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8d200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f88075390 0x7f4f8819f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59408/0 (socket says 172.21.15.84:59408) 2024-04-28T12:48:53.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8d200640 1 -- 172.21.15.84:0/3466700769 learned_addr learned my addr 172.21.15.84:0/3466700769 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:53.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f87e00640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f88076970 0x7f4f8819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:53.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8dc00640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f8810ee90 0x7f4f881a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:53.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8d200640 1 -- 172.21.15.84:0/3466700769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f88076970 msgr2=0x7f4f8819f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:53.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8d200640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f88076970 0x7f4f8819f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8d200640 1 -- 172.21.15.84:0/3466700769 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f8810ee90 msgr2=0x7f4f881a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:53.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8d200640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f8810ee90 0x7f4f881a6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.247+0000 7f4f8d200640 1 -- 172.21.15.84:0/3466700769 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f881a6c70 con 0x7f4f88075390 2024-04-28T12:48:53.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.248+0000 7f4f8d200640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f88075390 0x7f4f8819f050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800e9f0 tx=0x7f4f7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:53.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.248+0000 7f4f85600640 1 -- 172.21.15.84:0/3466700769 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7800cd80 con 0x7f4f88075390 2024-04-28T12:48:53.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.248+0000 7f4f85600640 1 -- 172.21.15.84:0/3466700769 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f780108e0 con 0x7f4f88075390 2024-04-28T12:48:53.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.248+0000 7f4f8e600640 1 -- 172.21.15.84:0/3466700769 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f881a6f60 con 0x7f4f88075390 2024-04-28T12:48:53.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.248+0000 7f4f85600640 1 -- 172.21.15.84:0/3466700769 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78018980 con 0x7f4f88075390 2024-04-28T12:48:53.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.248+0000 7f4f8e600640 1 -- 172.21.15.84:0/3466700769 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f881a73c0 con 0x7f4f88075390 2024-04-28T12:48:53.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.249+0000 7f4f8e600640 1 -- 172.21.15.84:0/3466700769 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f880696a0 con 0x7f4f88075390 2024-04-28T12:48:53.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.251+0000 7f4f85600640 1 -- 172.21.15.84:0/3466700769 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4f78010430 con 0x7f4f88075390 2024-04-28T12:48:53.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.253+0000 7f4f85600640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4f6c0a3bb0 0x7f4f6c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:53.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.253+0000 7f4f85600640 1 -- 172.21.15.84:0/3466700769 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4f780d0970 con 0x7f4f88075390 2024-04-28T12:48:53.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.253+0000 7f4f87e00640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4f6c0a3bb0 0x7f4f6c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:53.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.254+0000 7f4f87e00640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4f6c0a3bb0 0x7f4f6c0a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4f881a0560 tx=0x7f4f70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:53.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.258+0000 7f4f85600640 1 -- 172.21.15.84:0/3466700769 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4f880696a0 con 0x7f4f88075390 2024-04-28T12:48:53.458 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.462+0000 7f4f8e600640 1 -- 172.21.15.84:0/3466700769 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4f88063530 con 0x7f4f88075390 2024-04-28T12:48:53.459 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.463+0000 7f4f85600640 1 -- 172.21.15.84:0/3466700769 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f4f7809ca60 con 0x7f4f88075390 2024-04-28T12:48:53.459 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T12:48:53.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.465+0000 7f4f8e600640 1 -- 172.21.15.84:0/3466700769 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4f6c0a3bb0 msgr2=0x7f4f6c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:53.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.465+0000 7f4f8e600640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4f6c0a3bb0 0x7f4f6c0a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4f881a0560 tx=0x7f4f70005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:53.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.465+0000 7f4f8e600640 1 -- 172.21.15.84:0/3466700769 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f88075390 msgr2=0x7f4f8819f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:53.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.465+0000 7f4f8e600640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f88075390 0x7f4f8819f050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800e9f0 tx=0x7f4f7800eeb0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.465+0000 7f4f8e600640 1 -- 172.21.15.84:0/3466700769 shutdown_connections 2024-04-28T12:48:53.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.465+0000 7f4f8e600640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4f6c0a3bb0 0x7f4f6c0a6070 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.465+0000 7f4f8e600640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f8810ee90 0x7f4f881a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.465+0000 7f4f8e600640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f88076970 0x7f4f8819f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.465+0000 7f4f8e600640 1 --2- 172.21.15.84:0/3466700769 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f88075390 0x7f4f8819f050 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.465+0000 7f4f8e600640 1 -- 172.21.15.84:0/3466700769 >> 172.21.15.84:0/3466700769 conn(0x7f4f880fde30 msgr2=0x7f4f88102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:53.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.466+0000 7f4f8e600640 1 -- 172.21.15.84:0/3466700769 shutdown_connections 2024-04-28T12:48:53.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.466+0000 7f4f8e600640 1 -- 172.21.15.84:0/3466700769 wait complete. 2024-04-28T12:48:53.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T12:48:53.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.692+0000 7f4a6e000640 1 -- 172.21.15.84:0/2966524549 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a68106740 msgr2=0x7f4a6810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:53.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.692+0000 7f4a6e000640 1 --2- 172.21.15.84:0/2966524549 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a68106740 0x7f4a6810cfd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4a60009900 tx=0x7f4a6002a3f0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.692+0000 7f4a6e000640 1 -- 172.21.15.84:0/2966524549 shutdown_connections 2024-04-28T12:48:53.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.692+0000 7f4a6e000640 1 --2- 172.21.15.84:0/2966524549 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a68106740 0x7f4a6810cfd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.692+0000 7f4a6e000640 1 --2- 172.21.15.84:0/2966524549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a68105d80 0x7f4a68106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.692+0000 7f4a6e000640 1 --2- 172.21.15.84:0/2966524549 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a68104b90 0x7f4a68104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.692+0000 7f4a6e000640 1 -- 172.21.15.84:0/2966524549 >> 172.21.15.84:0/2966524549 conn(0x7f4a68100120 msgr2=0x7f4a68102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:53.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.692+0000 7f4a6e000640 1 -- 172.21.15.84:0/2966524549 shutdown_connections 2024-04-28T12:48:53.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.692+0000 7f4a6e000640 1 -- 172.21.15.84:0/2966524549 wait complete. 2024-04-28T12:48:53.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.693+0000 7f4a6e000640 1 Processor -- start 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.693+0000 7f4a6e000640 1 -- start start 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.693+0000 7f4a6e000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a68104b90 0x7f4a6819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.693+0000 7f4a6e000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a68105d80 0x7f4a6819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.693+0000 7f4a6e000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a68106740 0x7f4a681a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.693+0000 7f4a6e000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4a68110710 con 0x7f4a68106740 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.693+0000 7f4a6e000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4a68110590 con 0x7f4a68105d80 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.693+0000 7f4a6e000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4a68110890 con 0x7f4a68104b90 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.694+0000 7f4a67e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a68106740 0x7f4a681a22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.694+0000 7f4a5ea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a68105d80 0x7f4a6819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.694+0000 7f4a67400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a68104b90 0x7f4a6819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.694+0000 7f4a5ea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a68105d80 0x7f4a6819b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:55854/0 (socket says 172.21.15.84:55854) 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.694+0000 7f4a67400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a68104b90 0x7f4a6819ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:53634/0 (socket says 172.21.15.84:53634) 2024-04-28T12:48:53.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.694+0000 7f4a5ea00640 1 -- 172.21.15.84:0/242641401 learned_addr learned my addr 172.21.15.84:0/242641401 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:53.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.694+0000 7f4a67400640 1 -- 172.21.15.84:0/242641401 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a68105d80 msgr2=0x7f4a6819b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:53.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.694+0000 7f4a67400640 1 --2- 172.21.15.84:0/242641401 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a68105d80 0x7f4a6819b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.694+0000 7f4a67400640 1 -- 172.21.15.84:0/242641401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a68106740 msgr2=0x7f4a681a22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:53.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.694+0000 7f4a67400640 1 --2- 172.21.15.84:0/242641401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a68106740 0x7f4a681a22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.694+0000 7f4a67400640 1 -- 172.21.15.84:0/242641401 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a681a2950 con 0x7f4a68104b90 2024-04-28T12:48:53.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.695+0000 7f4a67400640 1 --2- 172.21.15.84:0/242641401 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a68104b90 0x7f4a6819ad40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a58005fd0 tx=0x7f4a5800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:53.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.695+0000 7f4a64c00640 1 -- 172.21.15.84:0/242641401 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a58003970 con 0x7f4a68104b90 2024-04-28T12:48:53.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.695+0000 7f4a6e000640 1 -- 172.21.15.84:0/242641401 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a681a2c40 con 0x7f4a68104b90 2024-04-28T12:48:53.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.695+0000 7f4a64c00640 1 -- 172.21.15.84:0/242641401 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a58004020 con 0x7f4a68104b90 2024-04-28T12:48:53.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.695+0000 7f4a64c00640 1 -- 172.21.15.84:0/242641401 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a58010720 con 0x7f4a68104b90 2024-04-28T12:48:53.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.695+0000 7f4a6e000640 1 -- 172.21.15.84:0/242641401 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a681a3190 con 0x7f4a68104b90 2024-04-28T12:48:53.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.696+0000 7f4a6e000640 1 -- 172.21.15.84:0/242641401 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a6810be50 con 0x7f4a68104b90 2024-04-28T12:48:53.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.698+0000 7f4a64c00640 1 -- 172.21.15.84:0/242641401 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4a580041d0 con 0x7f4a68104b90 2024-04-28T12:48:53.696 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.699+0000 7f4a64c00640 1 --2- 172.21.15.84:0/242641401 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4a4c0a3c30 0x7f4a4c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:53.696 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.699+0000 7f4a64c00640 1 -- 172.21.15.84:0/242641401 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4a580d0ec0 con 0x7f4a68104b90 2024-04-28T12:48:53.696 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.700+0000 7f4a5ea00640 1 --2- 172.21.15.84:0/242641401 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4a4c0a3c30 0x7f4a4c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:53.696 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.700+0000 7f4a5ea00640 1 --2- 172.21.15.84:0/242641401 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4a4c0a3c30 0x7f4a4c0a60f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009c90 tx=0x7f4a50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:53.700 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.704+0000 7f4a64c00640 1 -- 172.21.15.84:0/242641401 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4a5809d600 con 0x7f4a68104b90 2024-04-28T12:48:53.906 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.909+0000 7f4a6e000640 1 -- 172.21.15.84:0/242641401 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4a680635d0 con 0x7f4a68104b90 2024-04-28T12:48:53.906 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.910+0000 7f4a64c00640 1 -- 172.21.15.84:0/242641401 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f4a5809cf00 con 0x7f4a68104b90 2024-04-28T12:48:53.907 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 64 2024-04-28T12:48:53.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.913+0000 7f4a6e000640 1 -- 172.21.15.84:0/242641401 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4a4c0a3c30 msgr2=0x7f4a4c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:53.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.913+0000 7f4a6e000640 1 --2- 172.21.15.84:0/242641401 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4a4c0a3c30 0x7f4a4c0a60f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009c90 tx=0x7f4a50005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:53.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.913+0000 7f4a6e000640 1 -- 172.21.15.84:0/242641401 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a68104b90 msgr2=0x7f4a6819ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:53.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.913+0000 7f4a6e000640 1 --2- 172.21.15.84:0/242641401 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a68104b90 0x7f4a6819ad40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a58005fd0 tx=0x7f4a5800ef90 comp rx=0 tx=0).stop 2024-04-28T12:48:53.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.913+0000 7f4a6e000640 1 -- 172.21.15.84:0/242641401 shutdown_connections 2024-04-28T12:48:53.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.913+0000 7f4a6e000640 1 --2- 172.21.15.84:0/242641401 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4a4c0a3c30 0x7f4a4c0a60f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.913+0000 7f4a6e000640 1 --2- 172.21.15.84:0/242641401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a68106740 0x7f4a681a22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.913+0000 7f4a6e000640 1 --2- 172.21.15.84:0/242641401 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a68105d80 0x7f4a6819b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.913+0000 7f4a6e000640 1 --2- 172.21.15.84:0/242641401 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a68104b90 0x7f4a6819ad40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:53.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.913+0000 7f4a6e000640 1 -- 172.21.15.84:0/242641401 >> 172.21.15.84:0/242641401 conn(0x7f4a68100120 msgr2=0x7f4a68109020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:53.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.913+0000 7f4a6e000640 1 -- 172.21.15.84:0/242641401 shutdown_connections 2024-04-28T12:48:53.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:53.913+0000 7f4a6e000640 1 -- 172.21.15.84:0/242641401 wait complete. 2024-04-28T12:48:53.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:53 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/192889666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:53.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:53 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3466700769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:48:53.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:53 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/192889666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:53.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:53 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3466700769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:48:53.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T12:48:53.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:53 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/192889666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:53.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:53 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3466700769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:48:54.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.136+0000 7f8d5ac00640 1 -- 172.21.15.84:0/754069849 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d54102560 msgr2=0x7f8d541029e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.136+0000 7f8d5ac00640 1 --2- 172.21.15.84:0/754069849 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d54102560 0x7f8d541029e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c009ca0 tx=0x7f8d4c02a940 comp rx=0 tx=0).stop 2024-04-28T12:48:54.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.136+0000 7f8d5ac00640 1 -- 172.21.15.84:0/754069849 shutdown_connections 2024-04-28T12:48:54.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.136+0000 7f8d5ac00640 1 --2- 172.21.15.84:0/754069849 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d54102560 0x7f8d541029e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.136+0000 7f8d5ac00640 1 --2- 172.21.15.84:0/754069849 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d54101bc0 0x7f8d54102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.136+0000 7f8d5ac00640 1 --2- 172.21.15.84:0/754069849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d54101260 0x7f8d54101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.136+0000 7f8d5ac00640 1 -- 172.21.15.84:0/754069849 >> 172.21.15.84:0/754069849 conn(0x7f8d540fcde0 msgr2=0x7f8d540ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:54.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.136+0000 7f8d5ac00640 1 -- 172.21.15.84:0/754069849 shutdown_connections 2024-04-28T12:48:54.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.136+0000 7f8d5ac00640 1 -- 172.21.15.84:0/754069849 wait complete. 2024-04-28T12:48:54.135 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.137+0000 7f8d5ac00640 1 Processor -- start 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d5ac00640 1 -- start start 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d5ac00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d54101260 0x7f8d5419ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d5ac00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d54101bc0 0x7f8d5419b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d5ac00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d54102560 0x7f8d541a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d5ac00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8d541106a0 con 0x7f8d54101bc0 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d5ac00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54110520 con 0x7f8d54102560 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d5ac00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54110820 con 0x7f8d54101260 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d53400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d54101bc0 0x7f8d5419b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d53e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d54101260 0x7f8d5419ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d53e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d54101260 0x7f8d5419ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:53652/0 (socket says 172.21.15.84:53652) 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d53400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d54101bc0 0x7f8d5419b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59432/0 (socket says 172.21.15.84:59432) 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d53e00640 1 -- 172.21.15.84:0/3650710202 learned_addr learned my addr 172.21.15.84:0/3650710202 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.138+0000 7f8d58c00640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d54102560 0x7f8d541a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:54.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.139+0000 7f8d53400640 1 -- 172.21.15.84:0/3650710202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d54101260 msgr2=0x7f8d5419ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.139+0000 7f8d53400640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d54101260 0x7f8d5419ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.139+0000 7f8d53400640 1 -- 172.21.15.84:0/3650710202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d54102560 msgr2=0x7f8d541a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.139+0000 7f8d53400640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d54102560 0x7f8d541a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.139+0000 7f8d53400640 1 -- 172.21.15.84:0/3650710202 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d541a29b0 con 0x7f8d54101bc0 2024-04-28T12:48:54.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.139+0000 7f8d53e00640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d54101260 0x7f8d5419ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:48:54.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.139+0000 7f8d53400640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d54101bc0 0x7f8d5419b2e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00e9f0 tx=0x7f8d3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:54.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.139+0000 7f8d50c00640 1 -- 172.21.15.84:0/3650710202 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c00ccc0 con 0x7f8d54101bc0 2024-04-28T12:48:54.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.139+0000 7f8d50c00640 1 -- 172.21.15.84:0/3650710202 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d3c005020 con 0x7f8d54101bc0 2024-04-28T12:48:54.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.139+0000 7f8d50c00640 1 -- 172.21.15.84:0/3650710202 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c004a00 con 0x7f8d54101bc0 2024-04-28T12:48:54.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.139+0000 7f8d5ac00640 1 -- 172.21.15.84:0/3650710202 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d541a2ca0 con 0x7f8d54101bc0 2024-04-28T12:48:54.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.139+0000 7f8d5ac00640 1 -- 172.21.15.84:0/3650710202 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d541a31a0 con 0x7f8d54101bc0 2024-04-28T12:48:54.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.141+0000 7f8d5ac00640 1 -- 172.21.15.84:0/3650710202 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d5410c240 con 0x7f8d54101bc0 2024-04-28T12:48:54.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.142+0000 7f8d50c00640 1 -- 172.21.15.84:0/3650710202 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8d3c004ba0 con 0x7f8d54101bc0 2024-04-28T12:48:54.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.145+0000 7f8d50c00640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8d380a3bb0 0x7f8d380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:54.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.145+0000 7f8d50c00640 1 -- 172.21.15.84:0/3650710202 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8d3c0d06b0 con 0x7f8d54101bc0 2024-04-28T12:48:54.141 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.145+0000 7f8d53e00640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8d380a3bb0 0x7f8d380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:54.142 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.146+0000 7f8d53e00640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8d380a3bb0 0x7f8d380a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8d5419c270 tx=0x7f8d44005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:54.146 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.149+0000 7f8d50c00640 1 -- 172.21.15.84:0/3650710202 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8d3c09cdf0 con 0x7f8d54101bc0 2024-04-28T12:48:54.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.351+0000 7f8d5ac00640 1 -- 172.21.15.84:0/3650710202 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8d540635d0 con 0x7f8d54101bc0 2024-04-28T12:48:54.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.351+0000 7f8d50c00640 1 -- 172.21.15.84:0/3650710202 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f8d3c09c6f0 con 0x7f8d54101bc0 2024-04-28T12:48:54.348 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T12:48:54.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.354+0000 7f8d5ac00640 1 -- 172.21.15.84:0/3650710202 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8d380a3bb0 msgr2=0x7f8d380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.354+0000 7f8d5ac00640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8d380a3bb0 0x7f8d380a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8d5419c270 tx=0x7f8d44005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:54.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.354+0000 7f8d5ac00640 1 -- 172.21.15.84:0/3650710202 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d54101bc0 msgr2=0x7f8d5419b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.354+0000 7f8d5ac00640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d54101bc0 0x7f8d5419b2e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00e9f0 tx=0x7f8d3c00eeb0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.354+0000 7f8d5ac00640 1 -- 172.21.15.84:0/3650710202 shutdown_connections 2024-04-28T12:48:54.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.354+0000 7f8d5ac00640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8d380a3bb0 0x7f8d380a6070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.354+0000 7f8d5ac00640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d54102560 0x7f8d541a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.354+0000 7f8d5ac00640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d54101bc0 0x7f8d5419b2e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.354+0000 7f8d5ac00640 1 --2- 172.21.15.84:0/3650710202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d54101260 0x7f8d5419ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.354+0000 7f8d5ac00640 1 -- 172.21.15.84:0/3650710202 >> 172.21.15.84:0/3650710202 conn(0x7f8d540fcde0 msgr2=0x7f8d540fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:54.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.354+0000 7f8d5ac00640 1 -- 172.21.15.84:0/3650710202 shutdown_connections 2024-04-28T12:48:54.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.354+0000 7f8d5ac00640 1 -- 172.21.15.84:0/3650710202 wait complete. 2024-04-28T12:48:54.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-28T12:48:54.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.569+0000 7f75b2200640 1 -- 172.21.15.84:0/1096351048 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75ac10a940 msgr2=0x7f75ac1111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.569+0000 7f75b2200640 1 --2- 172.21.15.84:0/1096351048 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75ac10a940 0x7f75ac1111d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009ca0 tx=0x7f75a402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.569+0000 7f75b2200640 1 -- 172.21.15.84:0/1096351048 shutdown_connections 2024-04-28T12:48:54.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.569+0000 7f75b2200640 1 --2- 172.21.15.84:0/1096351048 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75ac10a940 0x7f75ac1111d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.569+0000 7f75b2200640 1 --2- 172.21.15.84:0/1096351048 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75ac109f80 0x7f75ac10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.569+0000 7f75b2200640 1 --2- 172.21.15.84:0/1096351048 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75ac108d90 0x7f75ac1091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.569+0000 7f75b2200640 1 -- 172.21.15.84:0/1096351048 >> 172.21.15.84:0/1096351048 conn(0x7f75ac077d30 msgr2=0x7f75ac07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:54.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.569+0000 7f75b2200640 1 -- 172.21.15.84:0/1096351048 shutdown_connections 2024-04-28T12:48:54.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.569+0000 7f75b2200640 1 -- 172.21.15.84:0/1096351048 wait complete. 2024-04-28T12:48:54.568 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.570+0000 7f75b2200640 1 Processor -- start 2024-04-28T12:48:54.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.571+0000 7f75b2200640 1 -- start start 2024-04-28T12:48:54.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.571+0000 7f75b2200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75ac108d90 0x7f75ac19eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:54.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.571+0000 7f75b2200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75ac109f80 0x7f75ac19f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:54.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.571+0000 7f75b2200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75ac10a940 0x7f75ac1a6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:54.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.571+0000 7f75b2200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac1148f0 con 0x7f75ac10a940 2024-04-28T12:48:54.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.571+0000 7f75b2200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac114770 con 0x7f75ac109f80 2024-04-28T12:48:54.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.571+0000 7f75b2200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac114a70 con 0x7f75ac108d90 2024-04-28T12:48:54.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.571+0000 7f75b0e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75ac108d90 0x7f75ac19eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:54.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.571+0000 7f75b0e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75ac108d90 0x7f75ac19eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:53670/0 (socket says 172.21.15.84:53670) 2024-04-28T12:48:54.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.571+0000 7f75b0e00640 1 -- 172.21.15.84:0/4037898710 learned_addr learned my addr 172.21.15.84:0/4037898710 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:54.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.571+0000 7f75b1800640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75ac10a940 0x7f75ac1a6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:54.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.571+0000 7f75abe00640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75ac109f80 0x7f75ac19f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:54.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.572+0000 7f75b0e00640 1 -- 172.21.15.84:0/4037898710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75ac109f80 msgr2=0x7f75ac19f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.572+0000 7f75b0e00640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75ac109f80 0x7f75ac19f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.572+0000 7f75b0e00640 1 -- 172.21.15.84:0/4037898710 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75ac10a940 msgr2=0x7f75ac1a6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.572+0000 7f75b0e00640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75ac10a940 0x7f75ac1a6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.572+0000 7f75b0e00640 1 -- 172.21.15.84:0/4037898710 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ac1a6af0 con 0x7f75ac108d90 2024-04-28T12:48:54.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.572+0000 7f75b0e00640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75ac108d90 0x7f75ac19eed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f75a000e9c0 tx=0x7f75a000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:54.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.572+0000 7f75a9600640 1 -- 172.21.15.84:0/4037898710 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a000cd80 con 0x7f75ac108d90 2024-04-28T12:48:54.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.572+0000 7f75a9600640 1 -- 172.21.15.84:0/4037898710 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75a00035f0 con 0x7f75ac108d90 2024-04-28T12:48:54.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.572+0000 7f75a9600640 1 -- 172.21.15.84:0/4037898710 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0010690 con 0x7f75ac108d90 2024-04-28T12:48:54.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.572+0000 7f75b2200640 1 -- 172.21.15.84:0/4037898710 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ac1a6de0 con 0x7f75ac108d90 2024-04-28T12:48:54.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.572+0000 7f75b2200640 1 -- 172.21.15.84:0/4037898710 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ac1a7190 con 0x7f75ac108d90 2024-04-28T12:48:54.571 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.573+0000 7f75b2200640 1 -- 172.21.15.84:0/4037898710 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75ac110050 con 0x7f75ac108d90 2024-04-28T12:48:54.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.576+0000 7f75a9600640 1 -- 172.21.15.84:0/4037898710 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f75a00037a0 con 0x7f75ac108d90 2024-04-28T12:48:54.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.577+0000 7f75a9600640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f75940a3c30 0x7f75940a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:54.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.577+0000 7f75a9600640 1 -- 172.21.15.84:0/4037898710 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f75a00d0a30 con 0x7f75ac108d90 2024-04-28T12:48:54.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.577+0000 7f75abe00640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f75940a3c30 0x7f75940a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:54.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.578+0000 7f75abe00640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f75940a3c30 0x7f75940a60f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f75ac1a03e0 tx=0x7f7598005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:54.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.579+0000 7f75a9600640 1 -- 172.21.15.84:0/4037898710 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f75a009d170 con 0x7f75ac108d90 2024-04-28T12:48:54.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.785+0000 7f75b2200640 1 -- 172.21.15.84:0/4037898710 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f75ac02dd80 con 0x7f75ac108d90 2024-04-28T12:48:54.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.786+0000 7f75a9600640 1 -- 172.21.15.84:0/4037898710 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+591 (secure 0 0 0) 0x7f75a009ca70 con 0x7f75ac108d90 2024-04-28T12:48:54.782 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/1611938555 2024-04-29T12:40:29.445750+0000 2024-04-28T12:48:54.783 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:6801/4213070051 2024-04-29T12:40:29.445750+0000 2024-04-28T12:48:54.783 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:6800/4213070051 2024-04-29T12:40:29.445750+0000 2024-04-28T12:48:54.783 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/173746096 2024-04-29T12:40:29.445750+0000 2024-04-28T12:48:54.783 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/3031983413 2024-04-29T12:39:55.311912+0000 2024-04-28T12:48:54.783 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/3332561455 2024-04-29T12:39:55.311912+0000 2024-04-28T12:48:54.783 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/3243457783 2024-04-29T12:39:55.311912+0000 2024-04-28T12:48:54.783 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:6800/4148140420 2024-04-29T12:39:55.311912+0000 2024-04-28T12:48:54.783 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/3787967310 2024-04-29T12:40:29.445750+0000 2024-04-28T12:48:54.783 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:6801/4148140420 2024-04-29T12:39:55.311912+0000 2024-04-28T12:48:54.784 INFO:teuthology.orchestra.run.smithi084.stderr:listed 10 entries 2024-04-28T12:48:54.785 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.788+0000 7f75b2200640 1 -- 172.21.15.84:0/4037898710 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f75940a3c30 msgr2=0x7f75940a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.785 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.788+0000 7f75b2200640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f75940a3c30 0x7f75940a60f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f75ac1a03e0 tx=0x7f7598005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:54.785 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.788+0000 7f75b2200640 1 -- 172.21.15.84:0/4037898710 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75ac108d90 msgr2=0x7f75ac19eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:54.785 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.788+0000 7f75b2200640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75ac108d90 0x7f75ac19eed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f75a000e9c0 tx=0x7f75a000ee80 comp rx=0 tx=0).stop 2024-04-28T12:48:54.785 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.788+0000 7f75b2200640 1 -- 172.21.15.84:0/4037898710 shutdown_connections 2024-04-28T12:48:54.785 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.788+0000 7f75b2200640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f75940a3c30 0x7f75940a60f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.785 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.788+0000 7f75b2200640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75ac10a940 0x7f75ac1a6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.785 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.788+0000 7f75b2200640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75ac109f80 0x7f75ac19f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.786 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.788+0000 7f75b2200640 1 --2- 172.21.15.84:0/4037898710 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75ac108d90 0x7f75ac19eed0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:54.786 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.788+0000 7f75b2200640 1 -- 172.21.15.84:0/4037898710 >> 172.21.15.84:0/4037898710 conn(0x7f75ac077d30 msgr2=0x7f75ac10d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:54.786 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.788+0000 7f75b2200640 1 -- 172.21.15.84:0/4037898710 shutdown_connections 2024-04-28T12:48:54.786 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:54.789+0000 7f75b2200640 1 -- 172.21.15.84:0/4037898710 wait complete. 2024-04-28T12:48:54.802 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:48:54.802 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T12:48:54.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:48:54.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:54 smithi179 ceph-mon[63029]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:54.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:54 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/242641401' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:48:54.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:54 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3650710202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:48:54.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:54 smithi084 ceph-mon[28081]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:54.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:54 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/242641401' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:48:54.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:54 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3650710202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:48:54.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:54 smithi161 ceph-mon[62387]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-28T12:48:54.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:54 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/242641401' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:48:54.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:54 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3650710202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:48:55.093 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.096+0000 7ff6fd000640 1 -- 172.21.15.84:0/1436940412 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f810a940 msgr2=0x7ff6f81111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:55.094 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.096+0000 7ff6fd000640 1 --2- 172.21.15.84:0/1436940412 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f810a940 0x7ff6f81111d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009ca0 tx=0x7ff6f002a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.094 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.096+0000 7ff6fd000640 1 -- 172.21.15.84:0/1436940412 shutdown_connections 2024-04-28T12:48:55.094 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.096+0000 7ff6fd000640 1 --2- 172.21.15.84:0/1436940412 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f810a940 0x7ff6f81111d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.094 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.096+0000 7ff6fd000640 1 --2- 172.21.15.84:0/1436940412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f8109f80 0x7ff6f810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.094 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.096+0000 7ff6fd000640 1 --2- 172.21.15.84:0/1436940412 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6f8108d90 0x7ff6f81091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.094 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.096+0000 7ff6fd000640 1 -- 172.21.15.84:0/1436940412 >> 172.21.15.84:0/1436940412 conn(0x7ff6f8077d30 msgr2=0x7ff6f807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:55.097 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.096+0000 7ff6fd000640 1 -- 172.21.15.84:0/1436940412 shutdown_connections 2024-04-28T12:48:55.097 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.096+0000 7ff6fd000640 1 -- 172.21.15.84:0/1436940412 wait complete. 2024-04-28T12:48:55.097 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.097+0000 7ff6fd000640 1 Processor -- start 2024-04-28T12:48:55.097 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.097+0000 7ff6fd000640 1 -- start start 2024-04-28T12:48:55.099 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6fd000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6f8108d90 0x7ff6f819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:55.099 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6fd000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f8109f80 0x7ff6f819f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:55.099 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6fd000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f810a940 0x7ff6f81a64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:55.099 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6fd000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f8114910 con 0x7ff6f8109f80 2024-04-28T12:48:55.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6fd000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f8114790 con 0x7ff6f810a940 2024-04-28T12:48:55.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6fd000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f8114a90 con 0x7ff6f8108d90 2024-04-28T12:48:55.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6eea00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f8109f80 0x7ff6f819f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:55.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6f7e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f810a940 0x7ff6f81a64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:55.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6f7e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f810a940 0x7ff6f81a64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:55914/0 (socket says 172.21.15.84:55914) 2024-04-28T12:48:55.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6f7e00640 1 -- 172.21.15.84:0/491169174 learned_addr learned my addr 172.21.15.84:0/491169174 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:55.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6f7400640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6f8108d90 0x7ff6f819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:55.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6f7e00640 1 -- 172.21.15.84:0/491169174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6f8108d90 msgr2=0x7ff6f819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:55.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6f7e00640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6f8108d90 0x7ff6f819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6f7e00640 1 -- 172.21.15.84:0/491169174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f8109f80 msgr2=0x7ff6f819f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:55.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6f7e00640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f8109f80 0x7ff6f819f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.098+0000 7ff6f7e00640 1 -- 172.21.15.84:0/491169174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f81a6b40 con 0x7ff6f810a940 2024-04-28T12:48:55.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.099+0000 7ff6f7e00640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f810a940 0x7ff6f81a64d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0033040 tx=0x7ff6f00316f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:55.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.099+0000 7ff6f4c00640 1 -- 172.21.15.84:0/491169174 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0003b70 con 0x7ff6f810a940 2024-04-28T12:48:55.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.099+0000 7ff6fd000640 1 -- 172.21.15.84:0/491169174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f81a6dd0 con 0x7ff6f810a940 2024-04-28T12:48:55.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.099+0000 7ff6f4c00640 1 -- 172.21.15.84:0/491169174 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6f0003d10 con 0x7ff6f810a940 2024-04-28T12:48:55.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.099+0000 7ff6f4c00640 1 -- 172.21.15.84:0/491169174 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f002fb40 con 0x7ff6f810a940 2024-04-28T12:48:55.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.099+0000 7ff6fd000640 1 -- 172.21.15.84:0/491169174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f81a7290 con 0x7ff6f810a940 2024-04-28T12:48:55.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.100+0000 7ff6fd000640 1 -- 172.21.15.84:0/491169174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6f81100b0 con 0x7ff6f810a940 2024-04-28T12:48:55.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.102+0000 7ff6f4c00640 1 -- 172.21.15.84:0/491169174 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff6f0004610 con 0x7ff6f810a940 2024-04-28T12:48:55.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.104+0000 7ff6f4c00640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff6dc0a3bb0 0x7ff6dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:55.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.104+0000 7ff6f4c00640 1 -- 172.21.15.84:0/491169174 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff6f00f1bc0 con 0x7ff6f810a940 2024-04-28T12:48:55.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.104+0000 7ff6f7400640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff6dc0a3bb0 0x7ff6dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:55.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.104+0000 7ff6f7400640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff6dc0a3bb0 0x7ff6dc0a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8109de0 tx=0x7ff6e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:55.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.106+0000 7ff6f4c00640 1 -- 172.21.15.84:0/491169174 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff6f00be300 con 0x7ff6f810a940 2024-04-28T12:48:55.303 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.307+0000 7ff6fd000640 1 -- 172.21.15.84:0/491169174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff6f802dcd0 con 0x7ff6f810a940 2024-04-28T12:48:55.305 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.309+0000 7ff6f4c00640 1 -- 172.21.15.84:0/491169174 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24117 (secure 0 0 0) 0x7ff6f00bdc00 con 0x7ff6f810a940 2024-04-28T12:48:55.306 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:48:55.306 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":90,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:48:40.059255+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:48:55.308 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.311+0000 7ff6fd000640 1 -- 172.21.15.84:0/491169174 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff6dc0a3bb0 msgr2=0x7ff6dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:55.308 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.311+0000 7ff6fd000640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff6dc0a3bb0 0x7ff6dc0a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8109de0 tx=0x7ff6e8005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:55.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.311+0000 7ff6fd000640 1 -- 172.21.15.84:0/491169174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f810a940 msgr2=0x7ff6f81a64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:55.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.311+0000 7ff6fd000640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f810a940 0x7ff6f81a64d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0033040 tx=0x7ff6f00316f0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.312+0000 7ff6fd000640 1 -- 172.21.15.84:0/491169174 shutdown_connections 2024-04-28T12:48:55.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.312+0000 7ff6fd000640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff6dc0a3bb0 0x7ff6dc0a6070 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.312+0000 7ff6fd000640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f810a940 0x7ff6f81a64d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.312+0000 7ff6fd000640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff6f8109f80 0x7ff6f819f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.312+0000 7ff6fd000640 1 --2- 172.21.15.84:0/491169174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff6f8108d90 0x7ff6f819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.312+0000 7ff6fd000640 1 -- 172.21.15.84:0/491169174 >> 172.21.15.84:0/491169174 conn(0x7ff6f8077d30 msgr2=0x7ff6f810d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:55.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.312+0000 7ff6fd000640 1 -- 172.21.15.84:0/491169174 shutdown_connections 2024-04-28T12:48:55.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.312+0000 7ff6fd000640 1 -- 172.21.15.84:0/491169174 wait complete. 2024-04-28T12:48:55.327 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:43:15.608080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T12:48:32.037809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T12:48:33.240166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T12:48:37.567238+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T12:48:55.328 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:48:55.580 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.583+0000 7f0feb200640 1 -- 172.21.15.84:0/2812818671 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fec10a980 msgr2=0x7f0fec1112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:55.580 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.583+0000 7f0feb200640 1 --2- 172.21.15.84:0/2812818671 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fec10a980 0x7f0fec1112a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009ca0 tx=0x7f0fe402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.580 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.583+0000 7f0feb200640 1 -- 172.21.15.84:0/2812818671 shutdown_connections 2024-04-28T12:48:55.580 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.583+0000 7f0feb200640 1 --2- 172.21.15.84:0/2812818671 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fec10a980 0x7f0fec1112a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.583+0000 7f0feb200640 1 --2- 172.21.15.84:0/2812818671 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fec109fc0 0x7f0fec10a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.583+0000 7f0feb200640 1 --2- 172.21.15.84:0/2812818671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fec108e70 0x7f0fec109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.583+0000 7f0feb200640 1 -- 172.21.15.84:0/2812818671 >> 172.21.15.84:0/2812818671 conn(0x7f0fec077e00 msgr2=0x7f0fec07a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:55.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.584+0000 7f0feb200640 1 -- 172.21.15.84:0/2812818671 shutdown_connections 2024-04-28T12:48:55.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.584+0000 7f0feb200640 1 -- 172.21.15.84:0/2812818671 wait complete. 2024-04-28T12:48:55.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.584+0000 7f0feb200640 1 Processor -- start 2024-04-28T12:48:55.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.585+0000 7f0feb200640 1 -- start start 2024-04-28T12:48:55.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.585+0000 7f0feb200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fec108e70 0x7f0fec19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:55.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.585+0000 7f0feb200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fec109fc0 0x7f0fec19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:55.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.585+0000 7f0feb200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fec10a980 0x7f0fec1a65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:55.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.585+0000 7f0feb200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f0fec1149d0 con 0x7f0fec109fc0 2024-04-28T12:48:55.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.585+0000 7f0feb200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0fec114850 con 0x7f0fec10a980 2024-04-28T12:48:55.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.585+0000 7f0feb200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f0fec114b50 con 0x7f0fec108e70 2024-04-28T12:48:55.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.585+0000 7f0fe9e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fec108e70 0x7f0fec19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:55.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.585+0000 7f0fe1400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fec109fc0 0x7f0fec19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:55.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.585+0000 7f0fe9e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fec108e70 0x7f0fec19eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:53694/0 (socket says 172.21.15.84:53694) 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.585+0000 7f0fe9e00640 1 -- 172.21.15.84:0/3529450015 learned_addr learned my addr 172.21.15.84:0/3529450015 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.585+0000 7f0fea800640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fec10a980 0x7f0fec1a65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.586+0000 7f0fe9e00640 1 -- 172.21.15.84:0/3529450015 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fec10a980 msgr2=0x7f0fec1a65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.586+0000 7f0fe9e00640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fec10a980 0x7f0fec1a65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.586+0000 7f0fe9e00640 1 -- 172.21.15.84:0/3529450015 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fec109fc0 msgr2=0x7f0fec19f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.586+0000 7f0fe9e00640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fec109fc0 0x7f0fec19f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.586+0000 7f0fe9e00640 1 -- 172.21.15.84:0/3529450015 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fec1a6c10 con 0x7f0fec108e70 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.586+0000 7f0fe9e00640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fec108e70 0x7f0fec19eff0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00d950 tx=0x7f0fdc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.586+0000 7f0fe3400640 1 -- 172.21.15.84:0/3529450015 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc017030 con 0x7f0fec108e70 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.586+0000 7f0fe3400640 1 -- 172.21.15.84:0/3529450015 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fdc004020 con 0x7f0fec108e70 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.586+0000 7f0feb200640 1 -- 172.21.15.84:0/3529450015 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fec1a6f00 con 0x7f0fec108e70 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.586+0000 7f0fe3400640 1 -- 172.21.15.84:0/3529450015 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc0047a0 con 0x7f0fec108e70 2024-04-28T12:48:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.586+0000 7f0feb200640 1 -- 172.21.15.84:0/3529450015 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fec1a7360 con 0x7f0fec108e70 2024-04-28T12:48:55.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.588+0000 7f0feb200640 1 -- 172.21.15.84:0/3529450015 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fec069ac0 con 0x7f0fec108e70 2024-04-28T12:48:55.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.590+0000 7f0fe3400640 1 -- 172.21.15.84:0/3529450015 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0fdc009a30 con 0x7f0fec108e70 2024-04-28T12:48:55.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.591+0000 7f0fe3400640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f0fd00a3bb0 0x7f0fd00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:55.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.591+0000 7f0fe3400640 1 -- 172.21.15.84:0/3529450015 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f0fdc0d01c0 con 0x7f0fec108e70 2024-04-28T12:48:55.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.591+0000 7f0fe1400640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f0fd00a3bb0 0x7f0fd00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:55.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.592+0000 7f0fe1400640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f0fd00a3bb0 0x7f0fd00a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0fec1a0500 tx=0x7f0fd4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:55.589 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.593+0000 7f0fe3400640 1 -- 172.21.15.84:0/3529450015 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0fdc09c900 con 0x7f0fec108e70 2024-04-28T12:48:55.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.800+0000 7f0feb200640 1 -- 172.21.15.84:0/3529450015 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0fec02db90 con 0x7f0fec108e70 2024-04-28T12:48:55.797 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.801+0000 7f0fe3400640 1 -- 172.21.15.84:0/3529450015 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f0fdc09c200 con 0x7f0fec108e70 2024-04-28T12:48:55.798 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-04-28T12:48:55.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.804+0000 7f0feb200640 1 -- 172.21.15.84:0/3529450015 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f0fd00a3bb0 msgr2=0x7f0fd00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:55.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.804+0000 7f0feb200640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f0fd00a3bb0 0x7f0fd00a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0fec1a0500 tx=0x7f0fd4005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:55.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.804+0000 7f0feb200640 1 -- 172.21.15.84:0/3529450015 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fec108e70 msgr2=0x7f0fec19eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:55.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.804+0000 7f0feb200640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fec108e70 0x7f0fec19eff0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00d950 tx=0x7f0fdc00de10 comp rx=0 tx=0).stop 2024-04-28T12:48:55.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.804+0000 7f0feb200640 1 -- 172.21.15.84:0/3529450015 shutdown_connections 2024-04-28T12:48:55.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.804+0000 7f0feb200640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f0fd00a3bb0 0x7f0fd00a6070 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.804+0000 7f0feb200640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fec10a980 0x7f0fec1a65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.804+0000 7f0feb200640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0fec109fc0 0x7f0fec19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.804+0000 7f0feb200640 1 --2- 172.21.15.84:0/3529450015 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0fec108e70 0x7f0fec19eff0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:55.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.804+0000 7f0feb200640 1 -- 172.21.15.84:0/3529450015 >> 172.21.15.84:0/3529450015 conn(0x7f0fec077e00 msgr2=0x7f0fec10d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:55.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.804+0000 7f0feb200640 1 -- 172.21.15.84:0/3529450015 shutdown_connections 2024-04-28T12:48:55.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:55.804+0000 7f0feb200640 1 -- 172.21.15.84:0/3529450015 wait complete. 2024-04-28T12:48:55.820 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T12:48:55.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:55 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4037898710' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T12:48:55.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:55 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/491169174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:55.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:55 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4037898710' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T12:48:55.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:55 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/491169174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:55.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:55 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4037898710' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T12:48:55.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:55 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/491169174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:48:56.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.030+0000 7f8ee9200640 1 -- 172.21.15.84:0/3180359780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ee410a980 msgr2=0x7f8ee41112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.030+0000 7f8ee9200640 1 --2- 172.21.15.84:0/3180359780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ee410a980 0x7f8ee41112a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8edc009ca0 tx=0x7f8edc02a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.030+0000 7f8ee9200640 1 -- 172.21.15.84:0/3180359780 shutdown_connections 2024-04-28T12:48:56.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.030+0000 7f8ee9200640 1 --2- 172.21.15.84:0/3180359780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ee410a980 0x7f8ee41112a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.030+0000 7f8ee9200640 1 --2- 172.21.15.84:0/3180359780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ee4109fc0 0x7f8ee410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.030+0000 7f8ee9200640 1 --2- 172.21.15.84:0/3180359780 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ee4108e70 0x7f8ee4109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.030+0000 7f8ee9200640 1 -- 172.21.15.84:0/3180359780 >> 172.21.15.84:0/3180359780 conn(0x7f8ee4077e00 msgr2=0x7f8ee407a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:56.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.031+0000 7f8ee9200640 1 -- 172.21.15.84:0/3180359780 shutdown_connections 2024-04-28T12:48:56.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.031+0000 7f8ee9200640 1 -- 172.21.15.84:0/3180359780 wait complete. 2024-04-28T12:48:56.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.031+0000 7f8ee9200640 1 Processor -- start 2024-04-28T12:48:56.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.032+0000 7f8ee9200640 1 -- start start 2024-04-28T12:48:56.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.032+0000 7f8ee9200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ee4108e70 0x7f8ee419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:56.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.032+0000 7f8ee9200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ee4109fc0 0x7f8ee419f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:56.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.032+0000 7f8ee9200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ee410a980 0x7f8ee41a6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:56.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.032+0000 7f8ee9200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee41149f0 con 0x7f8ee410a980 2024-04-28T12:48:56.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.032+0000 7f8ee9200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee4114870 con 0x7f8ee4109fc0 2024-04-28T12:48:56.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.032+0000 7f8ee9200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee4114b70 con 0x7f8ee4108e70 2024-04-28T12:48:56.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.032+0000 7f8ee3e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ee410a980 0x7f8ee41a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:56.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.032+0000 7f8ee3400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ee4108e70 0x7f8ee419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:56.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.032+0000 7f8ee3e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ee410a980 0x7f8ee41a6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59488/0 (socket says 172.21.15.84:59488) 2024-04-28T12:48:56.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.032+0000 7f8ee3e00640 1 -- 172.21.15.84:0/3665693973 learned_addr learned my addr 172.21.15.84:0/3665693973 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:56.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.032+0000 7f8edbe00640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ee4109fc0 0x7f8ee419f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:56.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.033+0000 7f8ee3e00640 1 -- 172.21.15.84:0/3665693973 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ee4108e70 msgr2=0x7f8ee419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.033+0000 7f8ee3e00640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ee4108e70 0x7f8ee419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.033+0000 7f8ee3e00640 1 -- 172.21.15.84:0/3665693973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ee4109fc0 msgr2=0x7f8ee419f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.033+0000 7f8ee3e00640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ee4109fc0 0x7f8ee419f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.033+0000 7f8ee3e00640 1 -- 172.21.15.84:0/3665693973 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee41a6bf0 con 0x7f8ee410a980 2024-04-28T12:48:56.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.033+0000 7f8ee3e00640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ee410a980 0x7f8ee41a6580 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8edc033040 tx=0x7f8edc0317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:56.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.033+0000 7f8ee0c00640 1 -- 172.21.15.84:0/3665693973 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8edc003b70 con 0x7f8ee410a980 2024-04-28T12:48:56.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.033+0000 7f8ee0c00640 1 -- 172.21.15.84:0/3665693973 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8edc003d10 con 0x7f8ee410a980 2024-04-28T12:48:56.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.033+0000 7f8ee0c00640 1 -- 172.21.15.84:0/3665693973 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8edc02fbd0 con 0x7f8ee410a980 2024-04-28T12:48:56.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.033+0000 7f8ee9200640 1 -- 172.21.15.84:0/3665693973 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee41a6e80 con 0x7f8ee410a980 2024-04-28T12:48:56.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.033+0000 7f8ee9200640 1 -- 172.21.15.84:0/3665693973 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee41a7340 con 0x7f8ee410a980 2024-04-28T12:48:56.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.035+0000 7f8ee9200640 1 -- 172.21.15.84:0/3665693973 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee4110120 con 0x7f8ee410a980 2024-04-28T12:48:56.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.036+0000 7f8ee0c00640 1 -- 172.21.15.84:0/3665693973 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8edc02fd70 con 0x7f8ee410a980 2024-04-28T12:48:56.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.038+0000 7f8ee0c00640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8ec80a3bb0 0x7f8ec80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:56.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.038+0000 7f8ee0c00640 1 -- 172.21.15.84:0/3665693973 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8edc0f1ae0 con 0x7f8ee410a980 2024-04-28T12:48:56.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.038+0000 7f8ee3400640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8ec80a3bb0 0x7f8ec80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:56.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.039+0000 7f8ee3400640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8ec80a3bb0 0x7f8ec80a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4009bd0 tx=0x7f8ed4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:56.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.042+0000 7f8ee0c00640 1 -- 172.21.15.84:0/3665693973 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8edc0be220 con 0x7f8ee410a980 2024-04-28T12:48:56.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.246+0000 7f8ee9200640 1 -- 172.21.15.84:0/3665693973 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8ee4063530 con 0x7f8ee410a980 2024-04-28T12:48:56.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.247+0000 7f8ee0c00640 1 -- 172.21.15.84:0/3665693973 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f8edc0bdb20 con 0x7f8ee410a980 2024-04-28T12:48:56.244 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T12:48:56.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.250+0000 7f8ee9200640 1 -- 172.21.15.84:0/3665693973 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8ec80a3bb0 msgr2=0x7f8ec80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.250+0000 7f8ee9200640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8ec80a3bb0 0x7f8ec80a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4009bd0 tx=0x7f8ed4005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:56.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.250+0000 7f8ee9200640 1 -- 172.21.15.84:0/3665693973 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ee410a980 msgr2=0x7f8ee41a6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.250+0000 7f8ee9200640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ee410a980 0x7f8ee41a6580 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8edc033040 tx=0x7f8edc0317c0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.250+0000 7f8ee9200640 1 -- 172.21.15.84:0/3665693973 shutdown_connections 2024-04-28T12:48:56.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.250+0000 7f8ee9200640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f8ec80a3bb0 0x7f8ec80a6070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.250+0000 7f8ee9200640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ee410a980 0x7f8ee41a6580 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.250+0000 7f8ee9200640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ee4109fc0 0x7f8ee419f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.250+0000 7f8ee9200640 1 --2- 172.21.15.84:0/3665693973 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ee4108e70 0x7f8ee419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.250+0000 7f8ee9200640 1 -- 172.21.15.84:0/3665693973 >> 172.21.15.84:0/3665693973 conn(0x7f8ee4077e00 msgr2=0x7f8ee410d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:56.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.251+0000 7f8ee9200640 1 -- 172.21.15.84:0/3665693973 shutdown_connections 2024-04-28T12:48:56.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.251+0000 7f8ee9200640 1 -- 172.21.15.84:0/3665693973 wait complete. 2024-04-28T12:48:56.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T12:48:56.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.476+0000 7fc9f1400640 1 -- 172.21.15.84:0/3196315384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9ec104990 msgr2=0x7fc9ec104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.476+0000 7fc9f1400640 1 --2- 172.21.15.84:0/3196315384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9ec104990 0x7fc9ec104db0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc009900 tx=0x7fc9dc02a3f0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.476+0000 7fc9f1400640 1 -- 172.21.15.84:0/3196315384 shutdown_connections 2024-04-28T12:48:56.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.476+0000 7fc9f1400640 1 --2- 172.21.15.84:0/3196315384 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9ec106540 0x7fc9ec10cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.476+0000 7fc9f1400640 1 --2- 172.21.15.84:0/3196315384 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9ec105b80 0x7fc9ec106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.476+0000 7fc9f1400640 1 --2- 172.21.15.84:0/3196315384 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9ec104990 0x7fc9ec104db0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.476+0000 7fc9f1400640 1 -- 172.21.15.84:0/3196315384 >> 172.21.15.84:0/3196315384 conn(0x7fc9ec0fff00 msgr2=0x7fc9ec102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:56.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.476+0000 7fc9f1400640 1 -- 172.21.15.84:0/3196315384 shutdown_connections 2024-04-28T12:48:56.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.476+0000 7fc9f1400640 1 -- 172.21.15.84:0/3196315384 wait complete. 2024-04-28T12:48:56.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.477+0000 7fc9f1400640 1 Processor -- start 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.477+0000 7fc9f1400640 1 -- start start 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9f1400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9ec104990 0x7fc9ec19abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9f1400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9ec105b80 0x7fc9ec19b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9f1400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9ec106540 0x7fc9ec1a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9f1400640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec110510 con 0x7fc9ec106540 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9f1400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec110390 con 0x7fc9ec105b80 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9f1400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec110690 con 0x7fc9ec104990 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9f0a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9ec106540 0x7fc9ec1a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9ebe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9ec104990 0x7fc9ec19abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9ebe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9ec104990 0x7fc9ec19abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:53740/0 (socket says 172.21.15.84:53740) 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9ebe00640 1 -- 172.21.15.84:0/2261409178 learned_addr learned my addr 172.21.15.84:0/2261409178 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9f0a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9ec106540 0x7fc9ec1a2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59518/0 (socket says 172.21.15.84:59518) 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9e3400640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9ec105b80 0x7fc9ec19b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9f0a00640 1 -- 172.21.15.84:0/2261409178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9ec104990 msgr2=0x7fc9ec19abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.478+0000 7fc9f0a00640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9ec104990 0x7fc9ec19abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.479+0000 7fc9f0a00640 1 -- 172.21.15.84:0/2261409178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9ec105b80 msgr2=0x7fc9ec19b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.479+0000 7fc9f0a00640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9ec105b80 0x7fc9ec19b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.479+0000 7fc9f0a00640 1 -- 172.21.15.84:0/2261409178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ec1a27d0 con 0x7fc9ec106540 2024-04-28T12:48:56.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.479+0000 7fc9f0a00640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9ec106540 0x7fc9ec1a2160 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009fd0 tx=0x7fc9e400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:56.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.479+0000 7fc9e9600640 1 -- 172.21.15.84:0/2261409178 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e400ce20 con 0x7fc9ec106540 2024-04-28T12:48:56.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.479+0000 7fc9f1400640 1 -- 172.21.15.84:0/2261409178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9ec1a2ac0 con 0x7fc9ec106540 2024-04-28T12:48:56.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.479+0000 7fc9e9600640 1 -- 172.21.15.84:0/2261409178 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9e4005020 con 0x7fc9ec106540 2024-04-28T12:48:56.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.479+0000 7fc9e9600640 1 -- 172.21.15.84:0/2261409178 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4018670 con 0x7fc9ec106540 2024-04-28T12:48:56.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.479+0000 7fc9f1400640 1 -- 172.21.15.84:0/2261409178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9ec1a3060 con 0x7fc9ec106540 2024-04-28T12:48:56.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.480+0000 7fc9f1400640 1 -- 172.21.15.84:0/2261409178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9ec10bd80 con 0x7fc9ec106540 2024-04-28T12:48:56.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.482+0000 7fc9e9600640 1 -- 172.21.15.84:0/2261409178 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc9e40054d0 con 0x7fc9ec106540 2024-04-28T12:48:56.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.484+0000 7fc9e9600640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc9d00a3bb0 0x7fc9d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:56.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.484+0000 7fc9e9600640 1 -- 172.21.15.84:0/2261409178 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc9e40d0de0 con 0x7fc9ec106540 2024-04-28T12:48:56.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.484+0000 7fc9ebe00640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc9d00a3bb0 0x7fc9d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:56.481 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.485+0000 7fc9ebe00640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc9d00a3bb0 0x7fc9d00a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc000c00 tx=0x7fc9dc005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:56.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.486+0000 7fc9e9600640 1 -- 172.21.15.84:0/2261409178 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc9e4053060 con 0x7fc9ec106540 2024-04-28T12:48:56.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.688+0000 7fc9f1400640 1 -- 172.21.15.84:0/2261409178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc9ec02dc60 con 0x7fc9ec106540 2024-04-28T12:48:56.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.689+0000 7fc9e9600640 1 -- 172.21.15.84:0/2261409178 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fc9e409d5b0 con 0x7fc9ec106540 2024-04-28T12:48:56.685 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 64 2024-04-28T12:48:56.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.691+0000 7fc9f1400640 1 -- 172.21.15.84:0/2261409178 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc9d00a3bb0 msgr2=0x7fc9d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.691+0000 7fc9f1400640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc9d00a3bb0 0x7fc9d00a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc000c00 tx=0x7fc9dc005a20 comp rx=0 tx=0).stop 2024-04-28T12:48:56.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.691+0000 7fc9f1400640 1 -- 172.21.15.84:0/2261409178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9ec106540 msgr2=0x7fc9ec1a2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.691+0000 7fc9f1400640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9ec106540 0x7fc9ec1a2160 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009fd0 tx=0x7fc9e400ef10 comp rx=0 tx=0).stop 2024-04-28T12:48:56.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.691+0000 7fc9f1400640 1 -- 172.21.15.84:0/2261409178 shutdown_connections 2024-04-28T12:48:56.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.691+0000 7fc9f1400640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc9d00a3bb0 0x7fc9d00a6070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.691+0000 7fc9f1400640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9ec106540 0x7fc9ec1a2160 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.691+0000 7fc9f1400640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9ec105b80 0x7fc9ec19b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.691+0000 7fc9f1400640 1 --2- 172.21.15.84:0/2261409178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9ec104990 0x7fc9ec19abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.691+0000 7fc9f1400640 1 -- 172.21.15.84:0/2261409178 >> 172.21.15.84:0/2261409178 conn(0x7fc9ec0fff00 msgr2=0x7fc9ec108f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:56.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.692+0000 7fc9f1400640 1 -- 172.21.15.84:0/2261409178 shutdown_connections 2024-04-28T12:48:56.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.692+0000 7fc9f1400640 1 -- 172.21.15.84:0/2261409178 wait complete. 2024-04-28T12:48:56.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T12:48:56.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.912+0000 7f24b5e00640 1 -- 172.21.15.84:0/1861760070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f24b01028c0 msgr2=0x7f24b0102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.912+0000 7f24b5e00640 1 --2- 172.21.15.84:0/1861760070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f24b01028c0 0x7f24b0102ce0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009ca0 tx=0x7f24a002a8c0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.913+0000 7f24b5e00640 1 -- 172.21.15.84:0/1861760070 shutdown_connections 2024-04-28T12:48:56.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.913+0000 7f24b5e00640 1 --2- 172.21.15.84:0/1861760070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24b0104470 0x7f24b010ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.913+0000 7f24b5e00640 1 --2- 172.21.15.84:0/1861760070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24b0103ab0 0x7f24b0103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.913+0000 7f24b5e00640 1 --2- 172.21.15.84:0/1861760070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f24b01028c0 0x7f24b0102ce0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.913+0000 7f24b5e00640 1 -- 172.21.15.84:0/1861760070 >> 172.21.15.84:0/1861760070 conn(0x7f24b00fde30 msgr2=0x7f24b0100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:56.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.913+0000 7f24b5e00640 1 -- 172.21.15.84:0/1861760070 shutdown_connections 2024-04-28T12:48:56.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.913+0000 7f24b5e00640 1 -- 172.21.15.84:0/1861760070 wait complete. 2024-04-28T12:48:56.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.914+0000 7f24b5e00640 1 Processor -- start 2024-04-28T12:48:56.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.914+0000 7f24b5e00640 1 -- start start 2024-04-28T12:48:56.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.914+0000 7f24b5e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f24b01028c0 0x7f24b019acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:56.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.914+0000 7f24b5e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24b0103ab0 0x7f24b019b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:56.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.914+0000 7f24b5e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24b0104470 0x7f24b01a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:56.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.914+0000 7f24b5e00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f24b010e380 con 0x7f24b01028c0 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.914+0000 7f24b5e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f24b010e200 con 0x7f24b0103ab0 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.914+0000 7f24b5e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f24b010e500 con 0x7f24b0104470 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.914+0000 7f24b4a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f24b01028c0 0x7f24b019acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.915+0000 7f24b4a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f24b01028c0 0x7f24b019acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59524/0 (socket says 172.21.15.84:59524) 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.915+0000 7f24b4a00640 1 -- 172.21.15.84:0/2690711898 learned_addr learned my addr 172.21.15.84:0/2690711898 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.915+0000 7f24b5400640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24b0104470 0x7f24b01a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.915+0000 7f24afe00640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24b0103ab0 0x7f24b019b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.915+0000 7f24b4a00640 1 -- 172.21.15.84:0/2690711898 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24b0104470 msgr2=0x7f24b01a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.915+0000 7f24b4a00640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24b0104470 0x7f24b01a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.915+0000 7f24b4a00640 1 -- 172.21.15.84:0/2690711898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24b0103ab0 msgr2=0x7f24b019b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.915+0000 7f24b4a00640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24b0103ab0 0x7f24b019b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.915+0000 7f24b4a00640 1 -- 172.21.15.84:0/2690711898 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b01a2900 con 0x7f24b01028c0 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.915+0000 7f24b4a00640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f24b01028c0 0x7f24b019acc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009430 tx=0x7f24a00316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:56.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.916+0000 7f24ad600640 1 -- 172.21.15.84:0/2690711898 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a0003b70 con 0x7f24b01028c0 2024-04-28T12:48:56.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.916+0000 7f24ad600640 1 -- 172.21.15.84:0/2690711898 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24a0003d10 con 0x7f24b01028c0 2024-04-28T12:48:56.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.916+0000 7f24b5e00640 1 -- 172.21.15.84:0/2690711898 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b01a2b90 con 0x7f24b01028c0 2024-04-28T12:48:56.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.916+0000 7f24ad600640 1 -- 172.21.15.84:0/2690711898 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a002fb60 con 0x7f24b01028c0 2024-04-28T12:48:56.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.916+0000 7f24b5e00640 1 -- 172.21.15.84:0/2690711898 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b01a3130 con 0x7f24b01028c0 2024-04-28T12:48:56.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.917+0000 7f24b5e00640 1 -- 172.21.15.84:0/2690711898 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b01a3300 con 0x7f24b01028c0 2024-04-28T12:48:56.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:56 smithi179 ceph-mon[63029]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T12:48:56.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:56 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3529450015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:56.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:56 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3665693973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:48:56.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.919+0000 7f24ad600640 1 -- 172.21.15.84:0/2690711898 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f24a002fd00 con 0x7f24b01028c0 2024-04-28T12:48:56.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:56 smithi084 ceph-mon[28081]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T12:48:56.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:56 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3529450015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:56.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:56 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3665693973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:48:56.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.920+0000 7f24ad600640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f24940a3c30 0x7f24940a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:56.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.920+0000 7f24ad600640 1 -- 172.21.15.84:0/2690711898 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f24a00f1bb0 con 0x7f24b01028c0 2024-04-28T12:48:56.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.921+0000 7f24afe00640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f24940a3c30 0x7f24940a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:56.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.921+0000 7f24afe00640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f24940a3c30 0x7f24940a60f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f24b019c1c0 tx=0x7f2498005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:56.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:56.925+0000 7f24ad600640 1 -- 172.21.15.84:0/2690711898 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f24a00be2f0 con 0x7f24b01028c0 2024-04-28T12:48:56.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:56 smithi161 ceph-mon[62387]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T12:48:56.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:56 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3529450015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:48:56.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:56 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3665693973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:48:57.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.127+0000 7f24b5e00640 1 -- 172.21.15.84:0/2690711898 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f24b01a34d0 con 0x7f24b01028c0 2024-04-28T12:48:57.124 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.128+0000 7f24ad600640 1 -- 172.21.15.84:0/2690711898 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f24a00bdbf0 con 0x7f24b01028c0 2024-04-28T12:48:57.124 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T12:48:57.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.131+0000 7f24b5e00640 1 -- 172.21.15.84:0/2690711898 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f24940a3c30 msgr2=0x7f24940a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:57.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.131+0000 7f24b5e00640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f24940a3c30 0x7f24940a60f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f24b019c1c0 tx=0x7f2498005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:57.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.131+0000 7f24b5e00640 1 -- 172.21.15.84:0/2690711898 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f24b01028c0 msgr2=0x7f24b019acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:57.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.131+0000 7f24b5e00640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f24b01028c0 0x7f24b019acc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009430 tx=0x7f24a00316e0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.131+0000 7f24b5e00640 1 -- 172.21.15.84:0/2690711898 shutdown_connections 2024-04-28T12:48:57.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.131+0000 7f24b5e00640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f24940a3c30 0x7f24940a60f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.131+0000 7f24b5e00640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24b0104470 0x7f24b01a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.131+0000 7f24b5e00640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24b0103ab0 0x7f24b019b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.131+0000 7f24b5e00640 1 --2- 172.21.15.84:0/2690711898 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f24b01028c0 0x7f24b019acc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.131+0000 7f24b5e00640 1 -- 172.21.15.84:0/2690711898 >> 172.21.15.84:0/2690711898 conn(0x7f24b00fde30 msgr2=0x7f24b0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:57.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.131+0000 7f24b5e00640 1 -- 172.21.15.84:0/2690711898 shutdown_connections 2024-04-28T12:48:57.129 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.131+0000 7f24b5e00640 1 -- 172.21.15.84:0/2690711898 wait complete. 2024-04-28T12:48:57.147 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-28T12:48:57.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.358+0000 7f44b5800640 1 -- 172.21.15.84:0/1913526366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b0104520 msgr2=0x7f44b010adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:57.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.358+0000 7f44b5800640 1 --2- 172.21.15.84:0/1913526366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b0104520 0x7f44b010adb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009ca0 tx=0x7f44a802a8b0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.359+0000 7f44b5800640 1 -- 172.21.15.84:0/1913526366 shutdown_connections 2024-04-28T12:48:57.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.359+0000 7f44b5800640 1 --2- 172.21.15.84:0/1913526366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b0104520 0x7f44b010adb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.359+0000 7f44b5800640 1 --2- 172.21.15.84:0/1913526366 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44b0103b60 0x7f44b0103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.359+0000 7f44b5800640 1 --2- 172.21.15.84:0/1913526366 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b0102970 0x7f44b0102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.359+0000 7f44b5800640 1 -- 172.21.15.84:0/1913526366 >> 172.21.15.84:0/1913526366 conn(0x7f44b00fdee0 msgr2=0x7f44b0100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:57.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.359+0000 7f44b5800640 1 -- 172.21.15.84:0/1913526366 shutdown_connections 2024-04-28T12:48:57.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.359+0000 7f44b5800640 1 -- 172.21.15.84:0/1913526366 wait complete. 2024-04-28T12:48:57.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.360+0000 7f44b5800640 1 Processor -- start 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.360+0000 7f44b5800640 1 -- start start 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.360+0000 7f44b5800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b0102970 0x7f44b00766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44b5800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b0103b60 0x7f44b0074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44b5800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44b0104520 0x7f44b0075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44b5800640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f44b010e4f0 con 0x7f44b0104520 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44b5800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f44b010e370 con 0x7f44b0103b60 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44b5800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f44b010e670 con 0x7f44b0102970 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44aea00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b0102970 0x7f44b00766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44aea00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b0102970 0x7f44b00766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:53770/0 (socket says 172.21.15.84:53770) 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44aea00640 1 -- 172.21.15.84:0/2096032616 learned_addr learned my addr 172.21.15.84:0/2096032616 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44af400640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44b0104520 0x7f44b0075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44ae000640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b0103b60 0x7f44b0074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:57.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44aea00640 1 -- 172.21.15.84:0/2096032616 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b0103b60 msgr2=0x7f44b0074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:57.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44aea00640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b0103b60 0x7f44b0074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44aea00640 1 -- 172.21.15.84:0/2096032616 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44b0104520 msgr2=0x7f44b0075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:57.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44aea00640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44b0104520 0x7f44b0075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44aea00640 1 -- 172.21.15.84:0/2096032616 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b0075af0 con 0x7f44b0102970 2024-04-28T12:48:57.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.361+0000 7f44ae000640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b0103b60 0x7f44b0074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:48:57.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.362+0000 7f44aea00640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b0102970 0x7f44b00766e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f44a000e9c0 tx=0x7f44a000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:57.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.362+0000 7f44a7400640 1 -- 172.21.15.84:0/2096032616 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a000cd80 con 0x7f44b0102970 2024-04-28T12:48:57.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.362+0000 7f44a7400640 1 -- 172.21.15.84:0/2096032616 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44a00035f0 con 0x7f44b0102970 2024-04-28T12:48:57.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.362+0000 7f44a7400640 1 -- 172.21.15.84:0/2096032616 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0010690 con 0x7f44b0102970 2024-04-28T12:48:57.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.362+0000 7f44b5800640 1 -- 172.21.15.84:0/2096032616 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b01a2c80 con 0x7f44b0102970 2024-04-28T12:48:57.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.362+0000 7f44b5800640 1 -- 172.21.15.84:0/2096032616 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b01a3180 con 0x7f44b0102970 2024-04-28T12:48:57.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.363+0000 7f44b5800640 1 -- 172.21.15.84:0/2096032616 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44b0109c90 con 0x7f44b0102970 2024-04-28T12:48:57.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.365+0000 7f44a7400640 1 -- 172.21.15.84:0/2096032616 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f44a0004000 con 0x7f44b0102970 2024-04-28T12:48:57.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.366+0000 7f44a7400640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f44940a3bb0 0x7f44940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:48:57.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.367+0000 7f44a7400640 1 -- 172.21.15.84:0/2096032616 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f44a00d0db0 con 0x7f44b0102970 2024-04-28T12:48:57.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.367+0000 7f44ae000640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f44940a3bb0 0x7f44940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:48:57.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.368+0000 7f44ae000640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f44940a3bb0 0x7f44940a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f44b0076450 tx=0x7f449c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:48:57.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.369+0000 7f44a7400640 1 -- 172.21.15.84:0/2096032616 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f44a009d4f0 con 0x7f44b0102970 2024-04-28T12:48:57.578 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.582+0000 7f44b5800640 1 -- 172.21.15.84:0/2096032616 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f44b002dc90 con 0x7f44b0102970 2024-04-28T12:48:57.579 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.583+0000 7f44a7400640 1 -- 172.21.15.84:0/2096032616 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+591 (secure 0 0 0) 0x7f44a009cdf0 con 0x7f44b0102970 2024-04-28T12:48:57.579 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/1611938555 2024-04-29T12:40:29.445750+0000 2024-04-28T12:48:57.579 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:6801/4213070051 2024-04-29T12:40:29.445750+0000 2024-04-28T12:48:57.579 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:6800/4213070051 2024-04-29T12:40:29.445750+0000 2024-04-28T12:48:57.580 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/173746096 2024-04-29T12:40:29.445750+0000 2024-04-28T12:48:57.580 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/3031983413 2024-04-29T12:39:55.311912+0000 2024-04-28T12:48:57.580 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/3332561455 2024-04-29T12:39:55.311912+0000 2024-04-28T12:48:57.580 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/3243457783 2024-04-29T12:39:55.311912+0000 2024-04-28T12:48:57.580 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:6800/4148140420 2024-04-29T12:39:55.311912+0000 2024-04-28T12:48:57.580 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/3787967310 2024-04-29T12:40:29.445750+0000 2024-04-28T12:48:57.580 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:6801/4148140420 2024-04-29T12:39:55.311912+0000 2024-04-28T12:48:57.580 INFO:teuthology.orchestra.run.smithi084.stderr:listed 10 entries 2024-04-28T12:48:57.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.585+0000 7f44b5800640 1 -- 172.21.15.84:0/2096032616 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f44940a3bb0 msgr2=0x7f44940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:57.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.585+0000 7f44b5800640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f44940a3bb0 0x7f44940a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f44b0076450 tx=0x7f449c005f90 comp rx=0 tx=0).stop 2024-04-28T12:48:57.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.585+0000 7f44b5800640 1 -- 172.21.15.84:0/2096032616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b0102970 msgr2=0x7f44b00766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:48:57.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.585+0000 7f44b5800640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b0102970 0x7f44b00766e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f44a000e9c0 tx=0x7f44a000ee80 comp rx=0 tx=0).stop 2024-04-28T12:48:57.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.585+0000 7f44b5800640 1 -- 172.21.15.84:0/2096032616 shutdown_connections 2024-04-28T12:48:57.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.585+0000 7f44b5800640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f44940a3bb0 0x7f44940a6070 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.585+0000 7f44b5800640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44b0104520 0x7f44b0075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.585+0000 7f44b5800640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b0103b60 0x7f44b0074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.585+0000 7f44b5800640 1 --2- 172.21.15.84:0/2096032616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f44b0102970 0x7f44b00766e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:48:57.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.585+0000 7f44b5800640 1 -- 172.21.15.84:0/2096032616 >> 172.21.15.84:0/2096032616 conn(0x7f44b00fdee0 msgr2=0x7f44b0106e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:48:57.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.585+0000 7f44b5800640 1 -- 172.21.15.84:0/2096032616 shutdown_connections 2024-04-28T12:48:57.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:48:57.586+0000 7f44b5800640 1 -- 172.21.15.84:0/2096032616 wait complete. 2024-04-28T12:48:57.599 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi084.front.sepia.ceph.com... 2024-04-28T12:48:57.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:48:57.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-28T12:48:57.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi084.stderr:ceph-fuse[68641]: fuse finished with error 0 and tester_r 0 2024-04-28T12:48:57.682 INFO:teuthology.orchestra.run:waiting for 300 2024-04-28T12:48:57.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:57 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2261409178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:48:57.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:57 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2690711898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:48:57.857 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:57 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2096032616' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T12:48:57.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:57 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2261409178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:48:57.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:57 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2690711898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:48:57.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:57 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2096032616' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T12:48:57.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:57 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2261409178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:48:57.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:57 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2690711898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:48:57.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:57 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2096032616' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T12:48:58.914 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:48:58 smithi179 ceph-mon[63029]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T12:48:58.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:48:58 smithi084 ceph-mon[28081]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T12:48:58.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:48:58 smithi161 ceph-mon[62387]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T12:49:00.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:00 smithi179 ceph-mon[63029]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T12:49:00.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:00 smithi084 ceph-mon[28081]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T12:49:00.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:00 smithi161 ceph-mon[62387]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-28T12:49:02.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:02 smithi084 ceph-mon[28081]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T12:49:02.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:02 smithi161 ceph-mon[62387]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T12:49:03.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:02 smithi179 ceph-mon[63029]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T12:49:03.783 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi084.front.sepia.ceph.com 2024-04-28T12:49:03.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:49:03.784 DEBUG:teuthology.orchestra.run.smithi084:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-28T12:49:03.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:49:03.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:03 smithi161 ceph-mon[62387]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T12:49:04.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.044+0000 7fe9a3200640 1 -- 172.21.15.84:0/581786084 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a4100fe0 msgr2=0x7fe9a4101400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.044+0000 7fe9a3200640 1 --2- 172.21.15.84:0/581786084 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a4100fe0 0x7fe9a4101400 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe994009900 tx=0x7fe99402a3f0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.044+0000 7fe9a3200640 1 -- 172.21.15.84:0/581786084 shutdown_connections 2024-04-28T12:49:04.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.044+0000 7fe9a3200640 1 --2- 172.21.15.84:0/581786084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9a41022e0 0x7fe9a4102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.044+0000 7fe9a3200640 1 --2- 172.21.15.84:0/581786084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4101940 0x7fe9a4101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.044+0000 7fe9a3200640 1 --2- 172.21.15.84:0/581786084 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a4100fe0 0x7fe9a4101400 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.044+0000 7fe9a3200640 1 -- 172.21.15.84:0/581786084 >> 172.21.15.84:0/581786084 conn(0x7fe9a40fcbc0 msgr2=0x7fe9a40ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:04.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.044+0000 7fe9a3200640 1 -- 172.21.15.84:0/581786084 shutdown_connections 2024-04-28T12:49:04.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.044+0000 7fe9a3200640 1 -- 172.21.15.84:0/581786084 wait complete. 2024-04-28T12:49:04.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.045+0000 7fe9a3200640 1 Processor -- start 2024-04-28T12:49:04.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.045+0000 7fe9a3200640 1 -- start start 2024-04-28T12:49:04.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.046+0000 7fe9a3200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4100fe0 0x7fe9a419ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:04.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.046+0000 7fe9a3200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a4101940 0x7fe9a419b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:04.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.046+0000 7fe9a3200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9a41022e0 0x7fe9a41a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:04.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.046+0000 7fe9a3200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a41104d0 con 0x7fe9a4101940 2024-04-28T12:49:04.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.046+0000 7fe9a3200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a4110350 con 0x7fe9a4100fe0 2024-04-28T12:49:04.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.046+0000 7fe9a3200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a4110650 con 0x7fe9a41022e0 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.046+0000 7fe9a1400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a4101940 0x7fe9a419b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.046+0000 7fe9a1e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4100fe0 0x7fe9a419ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.046+0000 7fe9a1400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a4101940 0x7fe9a419b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50914/0 (socket says 172.21.15.84:50914) 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.046+0000 7fe9a1400640 1 -- 172.21.15.84:0/3541111988 learned_addr learned my addr 172.21.15.84:0/3541111988 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.046+0000 7fe9a2800640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9a41022e0 0x7fe9a41a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.047+0000 7fe9a1e00640 1 -- 172.21.15.84:0/3541111988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9a41022e0 msgr2=0x7fe9a41a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.047+0000 7fe9a1e00640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9a41022e0 0x7fe9a41a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.047+0000 7fe9a1e00640 1 -- 172.21.15.84:0/3541111988 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a4101940 msgr2=0x7fe9a419b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.047+0000 7fe9a1e00640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a4101940 0x7fe9a419b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.047+0000 7fe9a1e00640 1 -- 172.21.15.84:0/3541111988 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a41a2760 con 0x7fe9a4100fe0 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.047+0000 7fe9a1400640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a4101940 0x7fe9a419b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.047+0000 7fe9a1e00640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4100fe0 0x7fe9a419ab40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe99402a9e0 tx=0x7fe994003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:04.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.047+0000 7fe99aa00640 1 -- 172.21.15.84:0/3541111988 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99403f070 con 0x7fe9a4100fe0 2024-04-28T12:49:04.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.047+0000 7fe9a3200640 1 -- 172.21.15.84:0/3541111988 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a41a29f0 con 0x7fe9a4100fe0 2024-04-28T12:49:04.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.047+0000 7fe99aa00640 1 -- 172.21.15.84:0/3541111988 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe994031c50 con 0x7fe9a4100fe0 2024-04-28T12:49:04.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.047+0000 7fe99aa00640 1 -- 172.21.15.84:0/3541111988 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe994035030 con 0x7fe9a4100fe0 2024-04-28T12:49:04.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.047+0000 7fe9a3200640 1 -- 172.21.15.84:0/3541111988 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a41a2f00 con 0x7fe9a4100fe0 2024-04-28T12:49:04.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.048+0000 7fe9a3200640 1 -- 172.21.15.84:0/3541111988 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a410bed0 con 0x7fe9a4100fe0 2024-04-28T12:49:04.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.051+0000 7fe99aa00640 1 -- 172.21.15.84:0/3541111988 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe994042070 con 0x7fe9a4100fe0 2024-04-28T12:49:04.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.052+0000 7fe99aa00640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe9840a3bb0 0x7fe9840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:04.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.052+0000 7fe99aa00640 1 -- 172.21.15.84:0/3541111988 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe9940f13c0 con 0x7fe9a4100fe0 2024-04-28T12:49:04.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.052+0000 7fe9a1400640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe9840a3bb0 0x7fe9840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:04.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.053+0000 7fe9a1400640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe9840a3bb0 0x7fe9840a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe9a419c050 tx=0x7fe98c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:04.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.055+0000 7fe99aa00640 1 -- 172.21.15.84:0/3541111988 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe9940bd4b0 con 0x7fe9a4100fe0 2024-04-28T12:49:04.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:03 smithi179 ceph-mon[63029]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T12:49:04.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:03 smithi084 ceph-mon[28081]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T12:49:04.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.257+0000 7fe9a3200640 1 -- 172.21.15.84:0/3541111988 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe9a4063560 con 0x7fe9a4100fe0 2024-04-28T12:49:04.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.259+0000 7fe99aa00640 1 -- 172.21.15.84:0/3541111988 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24117 (secure 0 0 0) 0x7fe9940c17b0 con 0x7fe9a4100fe0 2024-04-28T12:49:04.256 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:49:04.256 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":90,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:48:40.059255+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:49:04.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.262+0000 7fe9a3200640 1 -- 172.21.15.84:0/3541111988 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe9840a3bb0 msgr2=0x7fe9840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.262+0000 7fe9a3200640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe9840a3bb0 0x7fe9840a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe9a419c050 tx=0x7fe98c009500 comp rx=0 tx=0).stop 2024-04-28T12:49:04.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.262+0000 7fe9a3200640 1 -- 172.21.15.84:0/3541111988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4100fe0 msgr2=0x7fe9a419ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.262+0000 7fe9a3200640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4100fe0 0x7fe9a419ab40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe99402a9e0 tx=0x7fe994003c10 comp rx=0 tx=0).stop 2024-04-28T12:49:04.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.262+0000 7fe9a3200640 1 -- 172.21.15.84:0/3541111988 shutdown_connections 2024-04-28T12:49:04.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.262+0000 7fe9a3200640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe9840a3bb0 0x7fe9840a6070 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.262+0000 7fe9a3200640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9a41022e0 0x7fe9a41a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.262+0000 7fe9a3200640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a4101940 0x7fe9a419b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.262+0000 7fe9a3200640 1 --2- 172.21.15.84:0/3541111988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4100fe0 0x7fe9a419ab40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.262+0000 7fe9a3200640 1 -- 172.21.15.84:0/3541111988 >> 172.21.15.84:0/3541111988 conn(0x7fe9a40fcbc0 msgr2=0x7fe9a40fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:04.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.262+0000 7fe9a3200640 1 -- 172.21.15.84:0/3541111988 shutdown_connections 2024-04-28T12:49:04.260 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.262+0000 7fe9a3200640 1 -- 172.21.15.84:0/3541111988 wait complete. 2024-04-28T12:49:04.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:49:04.484 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.488+0000 7fbfa1000640 1 -- 172.21.15.84:0/3732795987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf9c109f80 msgr2=0x7fbf9c10a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.484 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.488+0000 7fbfa1000640 1 --2- 172.21.15.84:0/3732795987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf9c109f80 0x7fbf9c10a400 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbf88009900 tx=0x7fbf8802a3f0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.484 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.488+0000 7fbfa1000640 1 -- 172.21.15.84:0/3732795987 shutdown_connections 2024-04-28T12:49:04.485 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.488+0000 7fbfa1000640 1 --2- 172.21.15.84:0/3732795987 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf9c10a940 0x7fbf9c1111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.485 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.488+0000 7fbfa1000640 1 --2- 172.21.15.84:0/3732795987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf9c109f80 0x7fbf9c10a400 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.485 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.488+0000 7fbfa1000640 1 --2- 172.21.15.84:0/3732795987 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf9c108d90 0x7fbf9c1091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.485 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.488+0000 7fbfa1000640 1 -- 172.21.15.84:0/3732795987 >> 172.21.15.84:0/3732795987 conn(0x7fbf9c077d30 msgr2=0x7fbf9c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:04.485 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.488+0000 7fbfa1000640 1 -- 172.21.15.84:0/3732795987 shutdown_connections 2024-04-28T12:49:04.485 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.488+0000 7fbfa1000640 1 -- 172.21.15.84:0/3732795987 wait complete. 2024-04-28T12:49:04.486 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.489+0000 7fbfa1000640 1 Processor -- start 2024-04-28T12:49:04.486 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.489+0000 7fbfa1000640 1 -- start start 2024-04-28T12:49:04.486 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.489+0000 7fbfa1000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf9c108d90 0x7fbf9c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:04.486 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.489+0000 7fbfa1000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf9c109f80 0x7fbf9c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:04.486 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.489+0000 7fbfa1000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf9c10a940 0x7fbf9c1a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:04.486 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.489+0000 7fbfa1000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c114910 con 0x7fbf9c10a940 2024-04-28T12:49:04.486 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.489+0000 7fbfa1000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c114790 con 0x7fbf9c108d90 2024-04-28T12:49:04.486 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.489+0000 7fbfa1000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c114a90 con 0x7fbf9c109f80 2024-04-28T12:49:04.487 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.490+0000 7fbf9b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf9c108d90 0x7fbf9c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.490+0000 7fbf9b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf9c108d90 0x7fbf9c19efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:37462/0 (socket says 172.21.15.84:37462) 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.490+0000 7fbf9b400640 1 -- 172.21.15.84:0/287340875 learned_addr learned my addr 172.21.15.84:0/287340875 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.490+0000 7fbf9be00640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf9c10a940 0x7fbf9c1a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.490+0000 7fbf9aa00640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf9c109f80 0x7fbf9c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.490+0000 7fbf9be00640 1 -- 172.21.15.84:0/287340875 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf9c109f80 msgr2=0x7fbf9c19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.490+0000 7fbf9be00640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf9c109f80 0x7fbf9c19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.490+0000 7fbf9be00640 1 -- 172.21.15.84:0/287340875 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf9c108d90 msgr2=0x7fbf9c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.490+0000 7fbf9be00640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf9c108d90 0x7fbf9c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.490+0000 7fbf9be00640 1 -- 172.21.15.84:0/287340875 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9c1a6c00 con 0x7fbf9c10a940 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.491+0000 7fbf9be00640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf9c10a940 0x7fbf9c1a6590 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400ec50 tx=0x7fbf9400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.491+0000 7fbf93e00640 1 -- 172.21.15.84:0/287340875 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94003b70 con 0x7fbf9c10a940 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.491+0000 7fbf93e00640 1 -- 172.21.15.84:0/287340875 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf94003d10 con 0x7fbf9c10a940 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.491+0000 7fbfa1000640 1 -- 172.21.15.84:0/287340875 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf9c1a6ef0 con 0x7fbf9c10a940 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.491+0000 7fbf93e00640 1 -- 172.21.15.84:0/287340875 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94010830 con 0x7fbf9c10a940 2024-04-28T12:49:04.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.491+0000 7fbfa1000640 1 -- 172.21.15.84:0/287340875 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf9c1a73b0 con 0x7fbf9c10a940 2024-04-28T12:49:04.489 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.492+0000 7fbfa1000640 1 -- 172.21.15.84:0/287340875 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf9c110100 con 0x7fbf9c10a940 2024-04-28T12:49:04.490 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.494+0000 7fbf93e00640 1 -- 172.21.15.84:0/287340875 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbf94003ec0 con 0x7fbf9c10a940 2024-04-28T12:49:04.492 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.496+0000 7fbf93e00640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbf800a3c30 0x7fbf800a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:04.492 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.496+0000 7fbf93e00640 1 -- 172.21.15.84:0/287340875 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fbf940d0b50 con 0x7fbf9c10a940 2024-04-28T12:49:04.492 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.496+0000 7fbf9b400640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbf800a3c30 0x7fbf800a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:04.493 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.497+0000 7fbf9b400640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbf800a3c30 0x7fbf800a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0058c0 tx=0x7fbf8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:04.496 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.500+0000 7fbf93e00640 1 -- 172.21.15.84:0/287340875 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbf9409d290 con 0x7fbf9c10a940 2024-04-28T12:49:04.704 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.707+0000 7fbfa1000640 1 -- 172.21.15.84:0/287340875 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf9c063560 con 0x7fbf9c10a940 2024-04-28T12:49:04.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.710+0000 7fbf93e00640 1 -- 172.21.15.84:0/287340875 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24117 (secure 0 0 0) 0x7fbf9409cb90 con 0x7fbf9c10a940 2024-04-28T12:49:04.707 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:49:04.707 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":90,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:48:40.059255+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:49:04.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.712+0000 7fbfa1000640 1 -- 172.21.15.84:0/287340875 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbf800a3c30 msgr2=0x7fbf800a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.712+0000 7fbfa1000640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbf800a3c30 0x7fbf800a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0058c0 tx=0x7fbf8c005f90 comp rx=0 tx=0).stop 2024-04-28T12:49:04.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.712+0000 7fbfa1000640 1 -- 172.21.15.84:0/287340875 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf9c10a940 msgr2=0x7fbf9c1a6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.712+0000 7fbfa1000640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf9c10a940 0x7fbf9c1a6590 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400ec50 tx=0x7fbf9400c6a0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.713+0000 7fbfa1000640 1 -- 172.21.15.84:0/287340875 shutdown_connections 2024-04-28T12:49:04.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.713+0000 7fbfa1000640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbf800a3c30 0x7fbf800a60f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.713+0000 7fbfa1000640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf9c10a940 0x7fbf9c1a6590 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.713+0000 7fbfa1000640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf9c109f80 0x7fbf9c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.713+0000 7fbfa1000640 1 --2- 172.21.15.84:0/287340875 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf9c108d90 0x7fbf9c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.713+0000 7fbfa1000640 1 -- 172.21.15.84:0/287340875 >> 172.21.15.84:0/287340875 conn(0x7fbf9c077d30 msgr2=0x7fbf9c10d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:04.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.713+0000 7fbfa1000640 1 -- 172.21.15.84:0/287340875 shutdown_connections 2024-04-28T12:49:04.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.713+0000 7fbfa1000640 1 -- 172.21.15.84:0/287340875 wait complete. 2024-04-28T12:49:04.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-28T12:49:04.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.933+0000 7fed2e000640 1 -- 172.21.15.84:0/965085750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed28105c60 msgr2=0x7fed281060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.933+0000 7fed2e000640 1 --2- 172.21.15.84:0/965085750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed28105c60 0x7fed281060e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fed10009900 tx=0x7fed1002a3f0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.933+0000 7fed2e000640 1 -- 172.21.15.84:0/965085750 shutdown_connections 2024-04-28T12:49:04.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.933+0000 7fed2e000640 1 --2- 172.21.15.84:0/965085750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed28106620 0x7fed2810ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.933+0000 7fed2e000640 1 --2- 172.21.15.84:0/965085750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed28105c60 0x7fed281060e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.933+0000 7fed2e000640 1 --2- 172.21.15.84:0/965085750 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed28104a70 0x7fed28104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.933+0000 7fed2e000640 1 -- 172.21.15.84:0/965085750 >> 172.21.15.84:0/965085750 conn(0x7fed280fffe0 msgr2=0x7fed28102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:04.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.933+0000 7fed2e000640 1 -- 172.21.15.84:0/965085750 shutdown_connections 2024-04-28T12:49:04.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.933+0000 7fed2e000640 1 -- 172.21.15.84:0/965085750 wait complete. 2024-04-28T12:49:04.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.934+0000 7fed2e000640 1 Processor -- start 2024-04-28T12:49:04.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.934+0000 7fed2e000640 1 -- start start 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2e000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed28104a70 0x7fed2819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2e000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed28105c60 0x7fed2819b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2e000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed28106620 0x7fed281a2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2e000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fed281105f0 con 0x7fed28106620 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2e000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fed28110470 con 0x7fed28105c60 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2e000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fed28110770 con 0x7fed28104a70 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2d600640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed28106620 0x7fed281a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2d600640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed28106620 0x7fed281a2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50958/0 (socket says 172.21.15.84:50958) 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2d600640 1 -- 172.21.15.84:0/1734671415 learned_addr learned my addr 172.21.15.84:0/1734671415 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed27e00640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed28105c60 0x7fed2819b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2cc00640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed28104a70 0x7fed2819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2d600640 1 -- 172.21.15.84:0/1734671415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed28104a70 msgr2=0x7fed2819ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2d600640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed28104a70 0x7fed2819ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.934 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2d600640 1 -- 172.21.15.84:0/1734671415 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed28105c60 msgr2=0x7fed2819b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:04.934 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2d600640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed28105c60 0x7fed2819b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:04.934 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2d600640 1 -- 172.21.15.84:0/1734671415 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed281a28b0 con 0x7fed28106620 2024-04-28T12:49:04.934 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.935+0000 7fed2d600640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed28106620 0x7fed281a2210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fed2000eb10 tx=0x7fed2000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:04.934 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.936+0000 7fed25600640 1 -- 172.21.15.84:0/1734671415 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed20004b20 con 0x7fed28106620 2024-04-28T12:49:04.934 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.936+0000 7fed25600640 1 -- 172.21.15.84:0/1734671415 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed20004cc0 con 0x7fed28106620 2024-04-28T12:49:04.934 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.936+0000 7fed2e000640 1 -- 172.21.15.84:0/1734671415 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed281a2ba0 con 0x7fed28106620 2024-04-28T12:49:04.934 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.936+0000 7fed25600640 1 -- 172.21.15.84:0/1734671415 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed20010670 con 0x7fed28106620 2024-04-28T12:49:04.934 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.936+0000 7fed2e000640 1 -- 172.21.15.84:0/1734671415 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed281a30c0 con 0x7fed28106620 2024-04-28T12:49:04.934 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.937+0000 7fed2e000640 1 -- 172.21.15.84:0/1734671415 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed2810bdb0 con 0x7fed28106620 2024-04-28T12:49:04.935 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.939+0000 7fed25600640 1 -- 172.21.15.84:0/1734671415 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fed20005020 con 0x7fed28106620 2024-04-28T12:49:04.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.941+0000 7fed25600640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed0c0a3bb0 0x7fed0c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:04.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.941+0000 7fed25600640 1 -- 172.21.15.84:0/1734671415 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fed200d0c70 con 0x7fed28106620 2024-04-28T12:49:04.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.941+0000 7fed2cc00640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed0c0a3bb0 0x7fed0c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:04.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.942+0000 7fed2cc00640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed0c0a3bb0 0x7fed0c0a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fed2819bb40 tx=0x7fed18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:04.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:04.945+0000 7fed25600640 1 -- 172.21.15.84:0/1734671415 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fed2009d3b0 con 0x7fed28106620 2024-04-28T12:49:04.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:04 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3541111988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:05.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:04 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3541111988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:05.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:04 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3541111988' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:05.167 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.171+0000 7fed2e000640 1 -- 172.21.15.84:0/1734671415 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fed2802dad0 con 0x7fed28106620 2024-04-28T12:49:05.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.698+0000 7fed25600640 1 -- 172.21.15.84:0/1734671415 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fed2009ccb0 con 0x7fed28106620 2024-04-28T12:49:05.700 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.703+0000 7fed2e000640 1 -- 172.21.15.84:0/1734671415 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed0c0a3bb0 msgr2=0x7fed0c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:05.700 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.703+0000 7fed2e000640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed0c0a3bb0 0x7fed0c0a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fed2819bb40 tx=0x7fed18005f90 comp rx=0 tx=0).stop 2024-04-28T12:49:05.700 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.703+0000 7fed2e000640 1 -- 172.21.15.84:0/1734671415 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed28106620 msgr2=0x7fed281a2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:05.700 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.703+0000 7fed2e000640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed28106620 0x7fed281a2210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fed2000eb10 tx=0x7fed2000c2d0 comp rx=0 tx=0).stop 2024-04-28T12:49:05.701 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.704+0000 7fed2e000640 1 -- 172.21.15.84:0/1734671415 shutdown_connections 2024-04-28T12:49:05.701 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.704+0000 7fed2e000640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed0c0a3bb0 0x7fed0c0a6070 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:05.701 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.704+0000 7fed2e000640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed28106620 0x7fed281a2210 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:05.701 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.704+0000 7fed2e000640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed28105c60 0x7fed2819b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:05.701 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.704+0000 7fed2e000640 1 --2- 172.21.15.84:0/1734671415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed28104a70 0x7fed2819ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:05.701 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.704+0000 7fed2e000640 1 -- 172.21.15.84:0/1734671415 >> 172.21.15.84:0/1734671415 conn(0x7fed280fffe0 msgr2=0x7fed28108f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:05.701 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.704+0000 7fed2e000640 1 -- 172.21.15.84:0/1734671415 shutdown_connections 2024-04-28T12:49:05.701 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.704+0000 7fed2e000640 1 -- 172.21.15.84:0/1734671415 wait complete. 2024-04-28T12:49:05.722 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-28T12:49:05.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-28T12:49:05.936 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.939+0000 7fc9b9000640 1 -- 172.21.15.84:0/3577552592 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9b4075390 msgr2=0x7fc9b4075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:05.936 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.939+0000 7fc9b9000640 1 --2- 172.21.15.84:0/3577552592 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9b4075390 0x7fc9b4075810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009930 tx=0x7fc9a002a440 comp rx=0 tx=0).stop 2024-04-28T12:49:05.936 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.939+0000 7fc9b9000640 1 -- 172.21.15.84:0/3577552592 shutdown_connections 2024-04-28T12:49:05.936 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.939+0000 7fc9b9000640 1 --2- 172.21.15.84:0/3577552592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9b410ee10 0x7fc9b4111200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:05.936 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.939+0000 7fc9b9000640 1 --2- 172.21.15.84:0/3577552592 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9b4075390 0x7fc9b4075810 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:05.936 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.939+0000 7fc9b9000640 1 --2- 172.21.15.84:0/3577552592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9b4076970 0x7fc9b4074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:05.936 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.939+0000 7fc9b9000640 1 -- 172.21.15.84:0/3577552592 >> 172.21.15.84:0/3577552592 conn(0x7fc9b40fddd0 msgr2=0x7fc9b41001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:05.936 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.939+0000 7fc9b9000640 1 -- 172.21.15.84:0/3577552592 shutdown_connections 2024-04-28T12:49:05.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.939+0000 7fc9b9000640 1 -- 172.21.15.84:0/3577552592 wait complete. 2024-04-28T12:49:05.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.940+0000 7fc9b9000640 1 Processor -- start 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.940+0000 7fc9b9000640 1 -- start start 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b9000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9b4075390 0x7fc9b419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b9000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9b4076970 0x7fc9b419f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b9000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9b410ee10 0x7fc9b41a6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b9000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b4114990 con 0x7fc9b4076970 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b9000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b4114810 con 0x7fc9b4075390 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b9000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b4114b10 con 0x7fc9b410ee10 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b2a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9b4076970 0x7fc9b419f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b3e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9b410ee10 0x7fc9b41a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b2a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9b4076970 0x7fc9b419f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50982/0 (socket says 172.21.15.84:50982) 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b2a00640 1 -- 172.21.15.84:0/793198299 learned_addr learned my addr 172.21.15.84:0/793198299 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b3400640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9b4075390 0x7fc9b419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b3e00640 1 -- 172.21.15.84:0/793198299 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9b4075390 msgr2=0x7fc9b419f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b3e00640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9b4075390 0x7fc9b419f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:05.939 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b3e00640 1 -- 172.21.15.84:0/793198299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9b4076970 msgr2=0x7fc9b419f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:05.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b3e00640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9b4076970 0x7fc9b419f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:05.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.941+0000 7fc9b3e00640 1 -- 172.21.15.84:0/793198299 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b41a6ce0 con 0x7fc9b410ee10 2024-04-28T12:49:05.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.942+0000 7fc9b3e00640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9b410ee10 0x7fc9b41a6670 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00ec50 tx=0x7fc9ac00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:05.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.942+0000 7fc9abe00640 1 -- 172.21.15.84:0/793198299 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac003b70 con 0x7fc9b410ee10 2024-04-28T12:49:05.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.942+0000 7fc9abe00640 1 -- 172.21.15.84:0/793198299 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9ac003d10 con 0x7fc9b410ee10 2024-04-28T12:49:05.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.942+0000 7fc9abe00640 1 -- 172.21.15.84:0/793198299 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac010830 con 0x7fc9b410ee10 2024-04-28T12:49:05.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.942+0000 7fc9b9000640 1 -- 172.21.15.84:0/793198299 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b41a6fd0 con 0x7fc9b410ee10 2024-04-28T12:49:05.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.942+0000 7fc9b9000640 1 -- 172.21.15.84:0/793198299 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b41a7440 con 0x7fc9b410ee10 2024-04-28T12:49:05.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.943+0000 7fc9b9000640 1 -- 172.21.15.84:0/793198299 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b4104fe0 con 0x7fc9b410ee10 2024-04-28T12:49:05.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.945+0000 7fc9abe00640 1 -- 172.21.15.84:0/793198299 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc9ac003ec0 con 0x7fc9b410ee10 2024-04-28T12:49:05.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.946+0000 7fc9abe00640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc9980a3bb0 0x7fc9980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:05.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.947+0000 7fc9abe00640 1 -- 172.21.15.84:0/793198299 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc9ac0d0bf0 con 0x7fc9b410ee10 2024-04-28T12:49:05.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.947+0000 7fc9b3400640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc9980a3bb0 0x7fc9980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:05.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.947+0000 7fc9b3400640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc9980a3bb0 0x7fc9980a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40095f0 tx=0x7fc9a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:05.947 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:05.951+0000 7fc9abe00640 1 -- 172.21.15.84:0/793198299 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc9ac09d330 con 0x7fc9b410ee10 2024-04-28T12:49:05.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:05 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/287340875' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:05.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:05 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1734671415' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-28T12:49:05.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:05 smithi161 ceph-mon[62387]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T12:49:06.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:05 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/287340875' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:06.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:05 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1734671415' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-28T12:49:06.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:05 smithi179 ceph-mon[63029]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T12:49:06.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:05 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/287340875' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:06.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:05 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1734671415' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-28T12:49:06.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:05 smithi084 ceph-mon[28081]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-28T12:49:06.190 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.194+0000 7fc9b9000640 1 -- 172.21.15.84:0/793198299 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fc9b4063530 con 0x7fc9b410ee10 2024-04-28T12:49:06.702 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.706+0000 7fc9abe00640 1 -- 172.21.15.84:0/793198299 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fc9ac09cc30 con 0x7fc9b410ee10 2024-04-28T12:49:06.705 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.709+0000 7fc9b9000640 1 -- 172.21.15.84:0/793198299 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc9980a3bb0 msgr2=0x7fc9980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:06.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.709+0000 7fc9b9000640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc9980a3bb0 0x7fc9980a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40095f0 tx=0x7fc9a4005f90 comp rx=0 tx=0).stop 2024-04-28T12:49:06.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.709+0000 7fc9b9000640 1 -- 172.21.15.84:0/793198299 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9b410ee10 msgr2=0x7fc9b41a6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:06.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.709+0000 7fc9b9000640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9b410ee10 0x7fc9b41a6670 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00ec50 tx=0x7fc9ac00c6a0 comp rx=0 tx=0).stop 2024-04-28T12:49:06.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.709+0000 7fc9b9000640 1 -- 172.21.15.84:0/793198299 shutdown_connections 2024-04-28T12:49:06.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.709+0000 7fc9b9000640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc9980a3bb0 0x7fc9980a6070 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:06.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.709+0000 7fc9b9000640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc9b410ee10 0x7fc9b41a6670 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:06.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.709+0000 7fc9b9000640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9b4076970 0x7fc9b419f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:06.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.709+0000 7fc9b9000640 1 --2- 172.21.15.84:0/793198299 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9b4075390 0x7fc9b419f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:06.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.709+0000 7fc9b9000640 1 -- 172.21.15.84:0/793198299 >> 172.21.15.84:0/793198299 conn(0x7fc9b40fddd0 msgr2=0x7fc9b41021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:06.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.709+0000 7fc9b9000640 1 -- 172.21.15.84:0/793198299 shutdown_connections 2024-04-28T12:49:06.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.709+0000 7fc9b9000640 1 -- 172.21.15.84:0/793198299 wait complete. 2024-04-28T12:49:06.732 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-28T12:49:06.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.921+0000 7f6a41200640 1 -- 172.21.15.84:0/52822688 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a3c105d40 msgr2=0x7f6a3c108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:06.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.921+0000 7f6a41200640 1 --2- 172.21.15.84:0/52822688 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a3c105d40 0x7f6a3c108130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6a28009900 tx=0x7f6a2802a3f0 comp rx=0 tx=0).stop 2024-04-28T12:49:06.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.921+0000 7f6a41200640 1 -- 172.21.15.84:0/52822688 shutdown_connections 2024-04-28T12:49:06.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.921+0000 7f6a41200640 1 --2- 172.21.15.84:0/52822688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c108670 0x7f6a3c10aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:06.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.921+0000 7f6a41200640 1 --2- 172.21.15.84:0/52822688 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a3c105d40 0x7f6a3c108130 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:06.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.921+0000 7f6a41200640 1 --2- 172.21.15.84:0/52822688 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c103410 0x7f6a3c105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:06.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.921+0000 7f6a41200640 1 -- 172.21.15.84:0/52822688 >> 172.21.15.84:0/52822688 conn(0x7f6a3c0fce00 msgr2=0x7f6a3c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:06.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.922+0000 7f6a41200640 1 -- 172.21.15.84:0/52822688 shutdown_connections 2024-04-28T12:49:06.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.922+0000 7f6a41200640 1 -- 172.21.15.84:0/52822688 wait complete. 2024-04-28T12:49:06.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.923+0000 7f6a41200640 1 Processor -- start 2024-04-28T12:49:06.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.923+0000 7f6a41200640 1 -- start start 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.923+0000 7f6a41200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c103410 0x7f6a3c19ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.923+0000 7f6a41200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c105d40 0x7f6a3c19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.923+0000 7f6a41200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a3c108670 0x7f6a3c1a2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.923+0000 7f6a41200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c10e4d0 con 0x7f6a3c108670 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.923+0000 7f6a41200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c10e350 con 0x7f6a3c103410 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.923+0000 7f6a41200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c10e650 con 0x7f6a3c105d40 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a3a800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a3c108670 0x7f6a3c1a2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a39400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c105d40 0x7f6a3c19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a39e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c103410 0x7f6a3c19ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a39400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c105d40 0x7f6a3c19b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:50592/0 (socket says 172.21.15.84:50592) 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a39400640 1 -- 172.21.15.84:0/874930165 learned_addr learned my addr 172.21.15.84:0/874930165 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a3a800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a3c108670 0x7f6a3c1a2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50994/0 (socket says 172.21.15.84:50994) 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a39400640 1 -- 172.21.15.84:0/874930165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c103410 msgr2=0x7f6a3c19ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:06.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a39400640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c103410 0x7f6a3c19ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:06.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a39400640 1 -- 172.21.15.84:0/874930165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a3c108670 msgr2=0x7f6a3c1a2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:06.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a39400640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a3c108670 0x7f6a3c1a2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:06.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a39400640 1 -- 172.21.15.84:0/874930165 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c1a2a70 con 0x7f6a3c105d40 2024-04-28T12:49:06.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a3a800640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a3c108670 0x7f6a3c1a2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:06.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.924+0000 7f6a39400640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c105d40 0x7f6a3c19b390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6a28006010 tx=0x7f6a280042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:06.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.925+0000 7f6a32a00640 1 -- 172.21.15.84:0/874930165 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2803f030 con 0x7f6a3c105d40 2024-04-28T12:49:06.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.925+0000 7f6a41200640 1 -- 172.21.15.84:0/874930165 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a3c1a2d00 con 0x7f6a3c105d40 2024-04-28T12:49:06.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.925+0000 7f6a32a00640 1 -- 172.21.15.84:0/874930165 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a28031910 con 0x7f6a3c105d40 2024-04-28T12:49:06.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.925+0000 7f6a32a00640 1 -- 172.21.15.84:0/874930165 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28035070 con 0x7f6a3c105d40 2024-04-28T12:49:06.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.925+0000 7f6a41200640 1 -- 172.21.15.84:0/874930165 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a3c1a31e0 con 0x7f6a3c105d40 2024-04-28T12:49:06.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.926+0000 7f6a41200640 1 -- 172.21.15.84:0/874930165 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a3c10a1c0 con 0x7f6a3c105d40 2024-04-28T12:49:06.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.928+0000 7f6a32a00640 1 -- 172.21.15.84:0/874930165 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6a28042030 con 0x7f6a3c105d40 2024-04-28T12:49:06.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.929+0000 7f6a32a00640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a200a3bb0 0x7f6a200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:06.926 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.930+0000 7f6a32a00640 1 -- 172.21.15.84:0/874930165 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f6a280f0e30 con 0x7f6a3c105d40 2024-04-28T12:49:06.926 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.930+0000 7f6a39e00640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a200a3bb0 0x7f6a200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:06.926 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.930+0000 7f6a39e00640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a200a3bb0 0x7f6a200a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009bd0 tx=0x7f6a2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:06.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:06.932+0000 7f6a32a00640 1 -- 172.21.15.84:0/874930165 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6a280bd570 con 0x7f6a3c105d40 2024-04-28T12:49:06.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:06 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1734671415' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-28T12:49:06.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:06 smithi161 ceph-mon[62387]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:49:06.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:06 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/793198299' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-28T12:49:06.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:06 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-28T12:49:07.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:06 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1734671415' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-28T12:49:07.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:06 smithi179 ceph-mon[63029]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:49:07.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:06 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/793198299' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-28T12:49:07.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:06 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-28T12:49:07.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:06 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1734671415' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-28T12:49:07.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:06 smithi084 ceph-mon[28081]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:49:07.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:06 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/793198299' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-28T12:49:07.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:06 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-28T12:49:07.173 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.177+0000 7f6a41200640 1 -- 172.21.15.84:0/874930165 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f6a3c02dad0 con 0x7f6a3c105d40 2024-04-28T12:49:07.715 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.719+0000 7f6a32a00640 1 -- 172.21.15.84:0/874930165 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f6a280bce70 con 0x7f6a3c105d40 2024-04-28T12:49:07.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.724+0000 7f6a41200640 1 -- 172.21.15.84:0/874930165 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a200a3bb0 msgr2=0x7f6a200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:07.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.724+0000 7f6a41200640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a200a3bb0 0x7f6a200a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009bd0 tx=0x7f6a2c005f90 comp rx=0 tx=0).stop 2024-04-28T12:49:07.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.724+0000 7f6a41200640 1 -- 172.21.15.84:0/874930165 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c105d40 msgr2=0x7f6a3c19b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:07.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.724+0000 7f6a41200640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c105d40 0x7f6a3c19b390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6a28006010 tx=0x7f6a280042f0 comp rx=0 tx=0).stop 2024-04-28T12:49:07.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.724+0000 7f6a41200640 1 -- 172.21.15.84:0/874930165 shutdown_connections 2024-04-28T12:49:07.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.724+0000 7f6a41200640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a200a3bb0 0x7f6a200a6070 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:07.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.724+0000 7f6a41200640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a3c108670 0x7f6a3c1a2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:07.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.724+0000 7f6a41200640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a3c105d40 0x7f6a3c19b390 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:07.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.724+0000 7f6a41200640 1 --2- 172.21.15.84:0/874930165 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a3c103410 0x7f6a3c19ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:07.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.724+0000 7f6a41200640 1 -- 172.21.15.84:0/874930165 >> 172.21.15.84:0/874930165 conn(0x7f6a3c0fce00 msgr2=0x7f6a3c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:07.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.724+0000 7f6a41200640 1 -- 172.21.15.84:0/874930165 shutdown_connections 2024-04-28T12:49:07.722 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.725+0000 7f6a41200640 1 -- 172.21.15.84:0/874930165 wait complete. 2024-04-28T12:49:07.750 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-28T12:49:07.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.940+0000 7fb260a00640 1 -- 172.21.15.84:0/3763284870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb25c106620 msgr2=0x7fb25c10ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:07.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.940+0000 7fb260a00640 1 --2- 172.21.15.84:0/3763284870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb25c106620 0x7fb25c10ceb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb254009ca0 tx=0x7fb25402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:49:07.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.941+0000 7fb260a00640 1 -- 172.21.15.84:0/3763284870 shutdown_connections 2024-04-28T12:49:07.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.941+0000 7fb260a00640 1 --2- 172.21.15.84:0/3763284870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb25c106620 0x7fb25c10ceb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:07.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.941+0000 7fb260a00640 1 --2- 172.21.15.84:0/3763284870 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb25c105c60 0x7fb25c1060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:07.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.941+0000 7fb260a00640 1 --2- 172.21.15.84:0/3763284870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb25c104a70 0x7fb25c104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:07.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.941+0000 7fb260a00640 1 -- 172.21.15.84:0/3763284870 >> 172.21.15.84:0/3763284870 conn(0x7fb25c0fffe0 msgr2=0x7fb25c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:07.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.941+0000 7fb260a00640 1 -- 172.21.15.84:0/3763284870 shutdown_connections 2024-04-28T12:49:07.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.941+0000 7fb260a00640 1 -- 172.21.15.84:0/3763284870 wait complete. 2024-04-28T12:49:07.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb260a00640 1 Processor -- start 2024-04-28T12:49:07.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb260a00640 1 -- start start 2024-04-28T12:49:07.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb260a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb25c104a70 0x7fb25c19ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:07.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb260a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb25c105c60 0x7fb25c19b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:07.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb260a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb25c106620 0x7fb25c1a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:07.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb260a00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c1105d0 con 0x7fb25c105c60 2024-04-28T12:49:07.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb260a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c110450 con 0x7fb25c106620 2024-04-28T12:49:07.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb260a00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c110750 con 0x7fb25c104a70 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb259e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb25c105c60 0x7fb25c19b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb25b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb25c106620 0x7fb25c1a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb25a800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb25c104a70 0x7fb25c19ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb25b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb25c106620 0x7fb25c1a21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:37522/0 (socket says 172.21.15.84:37522) 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb259e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb25c105c60 0x7fb25c19b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:51018/0 (socket says 172.21.15.84:51018) 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.942+0000 7fb25b200640 1 -- 172.21.15.84:0/2266762888 learned_addr learned my addr 172.21.15.84:0/2266762888 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.943+0000 7fb25b200640 1 -- 172.21.15.84:0/2266762888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb25c104a70 msgr2=0x7fb25c19ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.943+0000 7fb25b200640 1 --2- 172.21.15.84:0/2266762888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb25c104a70 0x7fb25c19ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.943+0000 7fb25b200640 1 -- 172.21.15.84:0/2266762888 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb25c105c60 msgr2=0x7fb25c19b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.943+0000 7fb25b200640 1 --2- 172.21.15.84:0/2266762888 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb25c105c60 0x7fb25c19b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.943+0000 7fb25b200640 1 -- 172.21.15.84:0/2266762888 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25c1a2870 con 0x7fb25c106620 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.943+0000 7fb25b200640 1 --2- 172.21.15.84:0/2266762888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb25c106620 0x7fb25c1a21d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb254033040 tx=0x7fb2540317c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.943+0000 7fb253400640 1 -- 172.21.15.84:0/2266762888 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb254003b70 con 0x7fb25c106620 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.944+0000 7fb253400640 1 -- 172.21.15.84:0/2266762888 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb254003d10 con 0x7fb25c106620 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.944+0000 7fb260a00640 1 -- 172.21.15.84:0/2266762888 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb25c1a2b00 con 0x7fb25c106620 2024-04-28T12:49:07.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.944+0000 7fb253400640 1 -- 172.21.15.84:0/2266762888 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25402fbd0 con 0x7fb25c106620 2024-04-28T12:49:07.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.944+0000 7fb260a00640 1 -- 172.21.15.84:0/2266762888 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb25c1a3050 con 0x7fb25c106620 2024-04-28T12:49:07.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.945+0000 7fb260a00640 1 -- 172.21.15.84:0/2266762888 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb25c10bd90 con 0x7fb25c106620 2024-04-28T12:49:07.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.947+0000 7fb253400640 1 -- 172.21.15.84:0/2266762888 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb25402fd70 con 0x7fb25c106620 2024-04-28T12:49:07.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.948+0000 7fb253400640 1 --2- 172.21.15.84:0/2266762888 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb2400a3bb0 0x7fb2400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:07.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.948+0000 7fb253400640 1 -- 172.21.15.84:0/2266762888 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb2540f1d70 con 0x7fb25c106620 2024-04-28T12:49:07.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.948+0000 7fb25a800640 1 --2- 172.21.15.84:0/2266762888 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb2400a3bb0 0x7fb2400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:07.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.949+0000 7fb25a800640 1 --2- 172.21.15.84:0/2266762888 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb2400a3bb0 0x7fb2400a6070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb25c19c130 tx=0x7fb24c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:07.951 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:07.955+0000 7fb253400640 1 -- 172.21.15.84:0/2266762888 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb2540be4b0 con 0x7fb25c106620 2024-04-28T12:49:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:07 smithi161 ceph-mon[62387]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-28T12:49:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:07 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-28T12:49:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:07 smithi161 ceph-mon[62387]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-28T12:49:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:07 smithi161 ceph-mon[62387]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:49:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:07 smithi161 ceph-mon[62387]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-28T12:49:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:07 smithi161 ceph-mon[62387]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-28T12:49:07.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:07 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/874930165' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-28T12:49:07.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:07 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-28T12:49:07.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:07 smithi161 ceph-mon[62387]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-28T12:49:08.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:07 smithi179 ceph-mon[63029]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-28T12:49:08.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:07 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-28T12:49:08.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:07 smithi179 ceph-mon[63029]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-28T12:49:08.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:07 smithi179 ceph-mon[63029]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:49:08.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:07 smithi179 ceph-mon[63029]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-28T12:49:08.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:07 smithi179 ceph-mon[63029]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-28T12:49:08.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:07 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/874930165' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-28T12:49:08.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:07 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-28T12:49:08.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:07 smithi179 ceph-mon[63029]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-28T12:49:08.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:07 smithi084 ceph-mon[28081]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-28T12:49:08.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:07 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-28T12:49:08.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:07 smithi084 ceph-mon[28081]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-28T12:49:08.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:07 smithi084 ceph-mon[28081]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-28T12:49:08.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:07 smithi084 ceph-mon[28081]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-28T12:49:08.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:07 smithi084 ceph-mon[28081]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-28T12:49:08.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:07 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/874930165' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-28T12:49:08.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:07 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-28T12:49:08.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:07 smithi084 ceph-mon[28081]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-28T12:49:08.195 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.198+0000 7fb260a00640 1 -- 172.21.15.84:0/2266762888 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fb25c02dcf0 con 0x7fb25c106620 2024-04-28T12:49:08.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.733+0000 7fb253400640 1 -- 172.21.15.84:0/2266762888 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7fb25c02dcf0 con 0x7fb25c106620 2024-04-28T12:49:08.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.736+0000 7fb260a00640 1 -- 172.21.15.84:0/2266762888 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb2400a3bb0 msgr2=0x7fb2400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:08.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.736+0000 7fb260a00640 1 --2- 172.21.15.84:0/2266762888 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb2400a3bb0 0x7fb2400a6070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb25c19c130 tx=0x7fb24c00b390 comp rx=0 tx=0).stop 2024-04-28T12:49:08.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.736+0000 7fb260a00640 1 -- 172.21.15.84:0/2266762888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb25c106620 msgr2=0x7fb25c1a21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:08.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.736+0000 7fb260a00640 1 --2- 172.21.15.84:0/2266762888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb25c106620 0x7fb25c1a21d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb254033040 tx=0x7fb2540317c0 comp rx=0 tx=0).stop 2024-04-28T12:49:08.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.736+0000 7fb260a00640 1 -- 172.21.15.84:0/2266762888 shutdown_connections 2024-04-28T12:49:08.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.736+0000 7fb260a00640 1 --2- 172.21.15.84:0/2266762888 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb2400a3bb0 0x7fb2400a6070 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:08.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.736+0000 7fb260a00640 1 --2- 172.21.15.84:0/2266762888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb25c106620 0x7fb25c1a21d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:08.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.736+0000 7fb260a00640 1 --2- 172.21.15.84:0/2266762888 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb25c105c60 0x7fb25c19b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:08.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.736+0000 7fb260a00640 1 --2- 172.21.15.84:0/2266762888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb25c104a70 0x7fb25c19ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:08.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.736+0000 7fb260a00640 1 -- 172.21.15.84:0/2266762888 >> 172.21.15.84:0/2266762888 conn(0x7fb25c0fffe0 msgr2=0x7fb25c108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:08.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.737+0000 7fb260a00640 1 -- 172.21.15.84:0/2266762888 shutdown_connections 2024-04-28T12:49:08.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.737+0000 7fb260a00640 1 -- 172.21.15.84:0/2266762888 wait complete. 2024-04-28T12:49:08.759 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-28T12:49:08.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-28T12:49:08.966 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.970+0000 7fde0d000640 1 -- 172.21.15.84:0/792331969 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08078530 msgr2=0x7fde08078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:08.967 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.970+0000 7fde0d000640 1 --2- 172.21.15.84:0/792331969 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08078530 0x7fde08078990 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fddf0009900 tx=0x7fddf002a3f0 comp rx=0 tx=0).stop 2024-04-28T12:49:08.967 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.970+0000 7fde0d000640 1 -- 172.21.15.84:0/792331969 shutdown_connections 2024-04-28T12:49:08.967 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.970+0000 7fde0d000640 1 --2- 172.21.15.84:0/792331969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde08078ed0 0x7fde08079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:08.967 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.970+0000 7fde0d000640 1 --2- 172.21.15.84:0/792331969 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08078530 0x7fde08078990 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:08.967 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.970+0000 7fde0d000640 1 --2- 172.21.15.84:0/792331969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde080772f0 0x7fde08077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:08.967 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.970+0000 7fde0d000640 1 -- 172.21.15.84:0/792331969 >> 172.21.15.84:0/792331969 conn(0x7fde08100120 msgr2=0x7fde08102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:08.967 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.970+0000 7fde0d000640 1 -- 172.21.15.84:0/792331969 shutdown_connections 2024-04-28T12:49:08.967 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.970+0000 7fde0d000640 1 -- 172.21.15.84:0/792331969 wait complete. 2024-04-28T12:49:08.969 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.971+0000 7fde0d000640 1 Processor -- start 2024-04-28T12:49:08.969 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.971+0000 7fde0d000640 1 -- start start 2024-04-28T12:49:08.969 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.971+0000 7fde0d000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde080772f0 0x7fde0819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:08.969 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.971+0000 7fde0d000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08078530 0x7fde0819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:08.969 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.971+0000 7fde0d000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde08078ed0 0x7fde081a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:08.969 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.971+0000 7fde0d000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fde08114b30 con 0x7fde08078ed0 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.971+0000 7fde0d000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fde081149b0 con 0x7fde08078530 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.971+0000 7fde0d000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fde08114cb0 con 0x7fde080772f0 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.971+0000 7fddfd400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08078530 0x7fde0819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.971+0000 7fddfd400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08078530 0x7fde0819f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:43926/0 (socket says 172.21.15.84:43926) 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.972+0000 7fddfd400640 1 -- 172.21.15.84:0/403808848 learned_addr learned my addr 172.21.15.84:0/403808848 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.972+0000 7fde06800640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde08078ed0 0x7fde081a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.972+0000 7fde05e00640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde080772f0 0x7fde0819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.972+0000 7fddfd400640 1 -- 172.21.15.84:0/403808848 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde080772f0 msgr2=0x7fde0819f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.972+0000 7fddfd400640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde080772f0 0x7fde0819f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.972+0000 7fddfd400640 1 -- 172.21.15.84:0/403808848 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde08078ed0 msgr2=0x7fde081a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.972+0000 7fddfd400640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde08078ed0 0x7fde081a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.972+0000 7fddfd400640 1 -- 172.21.15.84:0/403808848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde081a6dc0 con 0x7fde08078530 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.972+0000 7fddfd400640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08078530 0x7fde0819f6e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fddf0005ab0 tx=0x7fddf0003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.972+0000 7fddff400640 1 -- 172.21.15.84:0/403808848 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf003f070 con 0x7fde08078530 2024-04-28T12:49:08.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.972+0000 7fddff400640 1 -- 172.21.15.84:0/403808848 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddf0031c50 con 0x7fde08078530 2024-04-28T12:49:08.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.973+0000 7fddff400640 1 -- 172.21.15.84:0/403808848 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf0035030 con 0x7fde08078530 2024-04-28T12:49:08.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.973+0000 7fde0d000640 1 -- 172.21.15.84:0/403808848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde081a7050 con 0x7fde08078530 2024-04-28T12:49:08.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.973+0000 7fde0d000640 1 -- 172.21.15.84:0/403808848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde081a7590 con 0x7fde08078530 2024-04-28T12:49:08.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.974+0000 7fde0d000640 1 -- 172.21.15.84:0/403808848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde08111520 con 0x7fde08078530 2024-04-28T12:49:08.972 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.976+0000 7fddff400640 1 -- 172.21.15.84:0/403808848 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fddf0042070 con 0x7fde08078530 2024-04-28T12:49:08.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.977+0000 7fddff400640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fddec0a3c30 0x7fddec0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:08.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.977+0000 7fddff400640 1 -- 172.21.15.84:0/403808848 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fddf00f1430 con 0x7fde08078530 2024-04-28T12:49:08.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.977+0000 7fde05e00640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fddec0a3c30 0x7fddec0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:08.974 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.978+0000 7fde05e00640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fddec0a3c30 0x7fddec0a60f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fddf80099a0 tx=0x7fddf8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:08.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:08.984+0000 7fddff400640 1 -- 172.21.15.84:0/403808848 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fddf00bd520 con 0x7fde08078530 2024-04-28T12:49:08.992 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 12:49:08 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b[92461]: 2024-04-28T12:49:08.717+0000 7f630a6ec700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-28T12:49:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:08 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-28T12:49:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:08 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] up:active 2024-04-28T12:49:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:08 smithi161 ceph-mon[62387]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-28T12:49:08.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:08 smithi161 ceph-mon[62387]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-28T12:49:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:08 smithi161 ceph-mon[62387]: Cluster is now healthy 2024-04-28T12:49:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:08 smithi161 ceph-mon[62387]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-28T12:49:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:08 smithi161 ceph-mon[62387]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:creating} 9 up:standby 2024-04-28T12:49:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:08 smithi161 ceph-mon[62387]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-28T12:49:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:08 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2266762888' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-28T12:49:08.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:08 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-28T12:49:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:08 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-28T12:49:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:08 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] up:active 2024-04-28T12:49:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:08 smithi179 ceph-mon[63029]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-28T12:49:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:08 smithi179 ceph-mon[63029]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-28T12:49:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:08 smithi179 ceph-mon[63029]: Cluster is now healthy 2024-04-28T12:49:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:08 smithi179 ceph-mon[63029]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-28T12:49:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:08 smithi179 ceph-mon[63029]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:creating} 9 up:standby 2024-04-28T12:49:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:08 smithi179 ceph-mon[63029]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-28T12:49:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:08 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2266762888' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-28T12:49:09.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:08 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-28T12:49:09.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:08 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-28T12:49:09.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:08 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] up:active 2024-04-28T12:49:09.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:08 smithi084 ceph-mon[28081]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-28T12:49:09.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:08 smithi084 ceph-mon[28081]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-28T12:49:09.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:08 smithi084 ceph-mon[28081]: Cluster is now healthy 2024-04-28T12:49:09.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:08 smithi084 ceph-mon[28081]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-28T12:49:09.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:08 smithi084 ceph-mon[28081]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:creating} 9 up:standby 2024-04-28T12:49:09.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:08 smithi084 ceph-mon[28081]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-28T12:49:09.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:08 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2266762888' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-28T12:49:09.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:08 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-28T12:49:09.209 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.213+0000 7fde0d000640 1 -- 172.21.15.84:0/403808848 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fde08062d70 con 0x7fddec0a3c30 2024-04-28T12:49:09.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.249+0000 7fddff400640 1 -- 172.21.15.84:0/403808848 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fde08062d70 con 0x7fddec0a3c30 2024-04-28T12:49:09.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.252+0000 7fde0d000640 1 -- 172.21.15.84:0/403808848 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fddec0a3c30 msgr2=0x7fddec0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.252+0000 7fde0d000640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fddec0a3c30 0x7fddec0a60f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fddf80099a0 tx=0x7fddf8009500 comp rx=0 tx=0).stop 2024-04-28T12:49:09.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.252+0000 7fde0d000640 1 -- 172.21.15.84:0/403808848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08078530 msgr2=0x7fde0819f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.252+0000 7fde0d000640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08078530 0x7fde0819f6e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fddf0005ab0 tx=0x7fddf0003b90 comp rx=0 tx=0).stop 2024-04-28T12:49:09.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.252+0000 7fde0d000640 1 -- 172.21.15.84:0/403808848 shutdown_connections 2024-04-28T12:49:09.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.252+0000 7fde0d000640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fddec0a3c30 0x7fddec0a60f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.252+0000 7fde0d000640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde08078ed0 0x7fde081a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.252+0000 7fde0d000640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08078530 0x7fde0819f6e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.252+0000 7fde0d000640 1 --2- 172.21.15.84:0/403808848 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fde080772f0 0x7fde0819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.252+0000 7fde0d000640 1 -- 172.21.15.84:0/403808848 >> 172.21.15.84:0/403808848 conn(0x7fde08100120 msgr2=0x7fde08101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:09.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.252+0000 7fde0d000640 1 -- 172.21.15.84:0/403808848 shutdown_connections 2024-04-28T12:49:09.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.252+0000 7fde0d000640 1 -- 172.21.15.84:0/403808848 wait complete. 2024-04-28T12:49:09.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-28T12:49:09.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.463+0000 7f07d1000640 1 -- 172.21.15.84:0/1163396449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07cc108f90 msgr2=0x7f07cc1093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.461 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.463+0000 7f07d1000640 1 --2- 172.21.15.84:0/1163396449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07cc108f90 0x7f07cc1093b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009900 tx=0x7f07bc02a410 comp rx=0 tx=0).stop 2024-04-28T12:49:09.461 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.464+0000 7f07d1000640 1 -- 172.21.15.84:0/1163396449 shutdown_connections 2024-04-28T12:49:09.461 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.464+0000 7f07d1000640 1 --2- 172.21.15.84:0/1163396449 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07cc10ab40 0x7f07cc1113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.461 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.464+0000 7f07d1000640 1 --2- 172.21.15.84:0/1163396449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07cc10a180 0x7f07cc10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.461 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.464+0000 7f07d1000640 1 --2- 172.21.15.84:0/1163396449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07cc108f90 0x7f07cc1093b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.461 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.464+0000 7f07d1000640 1 -- 172.21.15.84:0/1163396449 >> 172.21.15.84:0/1163396449 conn(0x7f07cc077f30 msgr2=0x7f07cc07a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:09.461 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.464+0000 7f07d1000640 1 -- 172.21.15.84:0/1163396449 shutdown_connections 2024-04-28T12:49:09.461 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.464+0000 7f07d1000640 1 -- 172.21.15.84:0/1163396449 wait complete. 2024-04-28T12:49:09.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.465+0000 7f07d1000640 1 Processor -- start 2024-04-28T12:49:09.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.465+0000 7f07d1000640 1 -- start start 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.466+0000 7f07d1000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07cc108f90 0x7f07cc19f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.466+0000 7f07d1000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07cc10a180 0x7f07cc19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.466+0000 7f07d1000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07cc10ab40 0x7f07cc1a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.466+0000 7f07d1000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc114b20 con 0x7f07cc10a180 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.466+0000 7f07d1000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc1149a0 con 0x7f07cc10ab40 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.466+0000 7f07d1000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f07cc114ca0 con 0x7f07cc108f90 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.466+0000 7f07c9400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07cc10a180 0x7f07cc19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.466+0000 7f07ca800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07cc10ab40 0x7f07cc1a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.466+0000 7f07c9e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07cc108f90 0x7f07cc19f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.466+0000 7f07ca800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07cc10ab40 0x7f07cc1a67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:43964/0 (socket says 172.21.15.84:43964) 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.466+0000 7f07ca800640 1 -- 172.21.15.84:0/758325253 learned_addr learned my addr 172.21.15.84:0/758325253 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.466+0000 7f07c9400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07cc10a180 0x7f07cc19f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43172/0 (socket says 172.21.15.84:43172) 2024-04-28T12:49:09.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.467+0000 7f07ca800640 1 -- 172.21.15.84:0/758325253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07cc108f90 msgr2=0x7f07cc19f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.467+0000 7f07ca800640 1 --2- 172.21.15.84:0/758325253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07cc108f90 0x7f07cc19f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.467+0000 7f07ca800640 1 -- 172.21.15.84:0/758325253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07cc10a180 msgr2=0x7f07cc19f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.467+0000 7f07ca800640 1 --2- 172.21.15.84:0/758325253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07cc10a180 0x7f07cc19f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.467+0000 7f07ca800640 1 -- 172.21.15.84:0/758325253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07cc1a6e20 con 0x7f07cc10ab40 2024-04-28T12:49:09.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.467+0000 7f07ca800640 1 --2- 172.21.15.84:0/758325253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07cc10ab40 0x7f07cc1a67b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f07c400eaa0 tx=0x7f07c400ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:09.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.467+0000 7f07c2a00640 1 -- 172.21.15.84:0/758325253 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c400cdb0 con 0x7f07cc10ab40 2024-04-28T12:49:09.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.467+0000 7f07c2a00640 1 -- 172.21.15.84:0/758325253 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07c4005020 con 0x7f07cc10ab40 2024-04-28T12:49:09.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.467+0000 7f07d1000640 1 -- 172.21.15.84:0/758325253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07cc1a7110 con 0x7f07cc10ab40 2024-04-28T12:49:09.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.467+0000 7f07c2a00640 1 -- 172.21.15.84:0/758325253 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c40116d0 con 0x7f07cc10ab40 2024-04-28T12:49:09.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.467+0000 7f07d1000640 1 -- 172.21.15.84:0/758325253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07cc1a7660 con 0x7f07cc10ab40 2024-04-28T12:49:09.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.468+0000 7f07d1000640 1 -- 172.21.15.84:0/758325253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07cc110380 con 0x7f07cc10ab40 2024-04-28T12:49:09.466 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.470+0000 7f07c2a00640 1 -- 172.21.15.84:0/758325253 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f07c40051d0 con 0x7f07cc10ab40 2024-04-28T12:49:09.468 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.472+0000 7f07c2a00640 1 --2- 172.21.15.84:0/758325253 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f07b00a3c30 0x7f07b00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.468 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.472+0000 7f07c2a00640 1 -- 172.21.15.84:0/758325253 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f07c40d0780 con 0x7f07cc10ab40 2024-04-28T12:49:09.468 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.472+0000 7f07c9e00640 1 --2- 172.21.15.84:0/758325253 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f07b00a3c30 0x7f07b00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.469 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.473+0000 7f07c9e00640 1 --2- 172.21.15.84:0/758325253 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f07b00a3c30 0x7f07b00a60f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f07bc005ab0 tx=0x7f07bc005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:09.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.480+0000 7f07c2a00640 1 -- 172.21.15.84:0/758325253 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f07c4016030 con 0x7f07cc10ab40 2024-04-28T12:49:09.666 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:49:09 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:49:09.223+0000 7f0172200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T12:49:09.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:49:09 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:49:09.223+0000 7f0172200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T12:49:09.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:49:09 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:49:09.223+0000 7f0172200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T12:49:09.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:49:09 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:49:09.223+0000 7f0172200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T12:49:09.667 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 12:49:09 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T12:49:09.223+0000 7f0172200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-28T12:49:09.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.699+0000 7f07d1000640 1 -- 172.21.15.84:0/758325253 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f07cc062d70 con 0x7f07b00a3c30 2024-04-28T12:49:09.736 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.739+0000 7f07c2a00640 1 -- 172.21.15.84:0/758325253 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f07cc062d70 con 0x7f07b00a3c30 2024-04-28T12:49:09.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.742+0000 7f07d1000640 1 -- 172.21.15.84:0/758325253 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f07b00a3c30 msgr2=0x7f07b00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.742+0000 7f07d1000640 1 --2- 172.21.15.84:0/758325253 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f07b00a3c30 0x7f07b00a60f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f07bc005ab0 tx=0x7f07bc005a20 comp rx=0 tx=0).stop 2024-04-28T12:49:09.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.742+0000 7f07d1000640 1 -- 172.21.15.84:0/758325253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07cc10ab40 msgr2=0x7f07cc1a67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.742+0000 7f07d1000640 1 --2- 172.21.15.84:0/758325253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07cc10ab40 0x7f07cc1a67b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f07c400eaa0 tx=0x7f07c400ef60 comp rx=0 tx=0).stop 2024-04-28T12:49:09.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.743+0000 7f07d1000640 1 -- 172.21.15.84:0/758325253 shutdown_connections 2024-04-28T12:49:09.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.743+0000 7f07d1000640 1 --2- 172.21.15.84:0/758325253 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f07b00a3c30 0x7f07b00a60f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.743+0000 7f07d1000640 1 --2- 172.21.15.84:0/758325253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07cc10ab40 0x7f07cc1a67b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.743+0000 7f07d1000640 1 --2- 172.21.15.84:0/758325253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07cc10a180 0x7f07cc19f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.743+0000 7f07d1000640 1 --2- 172.21.15.84:0/758325253 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f07cc108f90 0x7f07cc19f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.743+0000 7f07d1000640 1 -- 172.21.15.84:0/758325253 >> 172.21.15.84:0/758325253 conn(0x7f07cc077f30 msgr2=0x7f07cc10d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:09.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.743+0000 7f07d1000640 1 -- 172.21.15.84:0/758325253 shutdown_connections 2024-04-28T12:49:09.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.743+0000 7f07d1000640 1 -- 172.21.15.84:0/758325253 wait complete. 2024-04-28T12:49:09.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-28T12:49:09.951 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.954+0000 7f6a31600640 1 -- 172.21.15.84:0/3925147539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a2c105c60 msgr2=0x7f6a2c1060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.951 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.954+0000 7f6a31600640 1 --2- 172.21.15.84:0/3925147539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a2c105c60 0x7f6a2c1060e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009860 tx=0x7f6a1802a340 comp rx=0 tx=0).stop 2024-04-28T12:49:09.951 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.954+0000 7f6a31600640 1 -- 172.21.15.84:0/3925147539 shutdown_connections 2024-04-28T12:49:09.951 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.954+0000 7f6a31600640 1 --2- 172.21.15.84:0/3925147539 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a2c106620 0x7f6a2c10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.951 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.954+0000 7f6a31600640 1 --2- 172.21.15.84:0/3925147539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a2c105c60 0x7f6a2c1060e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.951 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.954+0000 7f6a31600640 1 --2- 172.21.15.84:0/3925147539 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a2c104a70 0x7f6a2c104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.951 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.954+0000 7f6a31600640 1 -- 172.21.15.84:0/3925147539 >> 172.21.15.84:0/3925147539 conn(0x7f6a2c0fffe0 msgr2=0x7f6a2c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:09.951 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.954+0000 7f6a31600640 1 -- 172.21.15.84:0/3925147539 shutdown_connections 2024-04-28T12:49:09.951 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.954+0000 7f6a31600640 1 -- 172.21.15.84:0/3925147539 wait complete. 2024-04-28T12:49:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.955+0000 7f6a31600640 1 Processor -- start 2024-04-28T12:49:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.955+0000 7f6a31600640 1 -- start start 2024-04-28T12:49:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a31600640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a2c104a70 0x7f6a2c19acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a31600640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a2c105c60 0x7f6a2c19b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a31600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a2c106620 0x7f6a2c1a2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a31600640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2c110580 con 0x7f6a2c105c60 2024-04-28T12:49:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a31600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2c110400 con 0x7f6a2c106620 2024-04-28T12:49:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a31600640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2c110700 con 0x7f6a2c104a70 2024-04-28T12:49:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a2b400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a2c105c60 0x7f6a2c19b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a2b400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a2c105c60 0x7f6a2c19b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43180/0 (socket says 172.21.15.84:43180) 2024-04-28T12:49:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a2b400640 1 -- 172.21.15.84:0/720491117 learned_addr learned my addr 172.21.15.84:0/720491117 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a2be00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a2c104a70 0x7f6a2c19acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a30c00640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a2c106620 0x7f6a2c1a2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a2b400640 1 -- 172.21.15.84:0/720491117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a2c104a70 msgr2=0x7f6a2c19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a2b400640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a2c104a70 0x7f6a2c19acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a2b400640 1 -- 172.21.15.84:0/720491117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a2c106620 msgr2=0x7f6a2c1a2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a2b400640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a2c106620 0x7f6a2c1a2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.956+0000 7f6a2b400640 1 -- 172.21.15.84:0/720491117 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2c1a28f0 con 0x7f6a2c105c60 2024-04-28T12:49:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.957+0000 7f6a2b400640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a2c105c60 0x7f6a2c19b1f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6a180057c0 tx=0x7f6a18032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.957+0000 7f6a28c00640 1 -- 172.21.15.84:0/720491117 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a18037040 con 0x7f6a2c105c60 2024-04-28T12:49:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.957+0000 7f6a28c00640 1 -- 172.21.15.84:0/720491117 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a18030bd0 con 0x7f6a2c105c60 2024-04-28T12:49:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.957+0000 7f6a31600640 1 -- 172.21.15.84:0/720491117 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a2c1a2b80 con 0x7f6a2c105c60 2024-04-28T12:49:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.957+0000 7f6a28c00640 1 -- 172.21.15.84:0/720491117 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a18005280 con 0x7f6a2c105c60 2024-04-28T12:49:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.957+0000 7f6a31600640 1 -- 172.21.15.84:0/720491117 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a2c1a3060 con 0x7f6a2c105c60 2024-04-28T12:49:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.959+0000 7f6a31600640 1 -- 172.21.15.84:0/720491117 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a2c10be80 con 0x7f6a2c105c60 2024-04-28T12:49:09.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.961+0000 7f6a28c00640 1 -- 172.21.15.84:0/720491117 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6a18043020 con 0x7f6a2c105c60 2024-04-28T12:49:09.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.962+0000 7f6a28c00640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a100a3c30 0x7f6a100a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:09.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.962+0000 7f6a28c00640 1 -- 172.21.15.84:0/720491117 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f6a180f01a0 con 0x7f6a2c105c60 2024-04-28T12:49:09.959 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.962+0000 7f6a2be00640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a100a3c30 0x7f6a100a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:09.959 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.963+0000 7f6a2be00640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a100a3c30 0x7f6a100a60f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c19bb80 tx=0x7f6a1c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:09.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:09.966+0000 7f6a28c00640 1 -- 172.21.15.84:0/720491117 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6a180bc960 con 0x7f6a2c105c60 2024-04-28T12:49:10.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:09 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-28T12:49:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:09 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] up:active 2024-04-28T12:49:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:09 smithi179 ceph-mon[63029]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 9 up:standby 2024-04-28T12:49:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:09 smithi179 ceph-mon[63029]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T12:49:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:09 smithi179 ceph-mon[63029]: from='client.24842 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:09 smithi179 ceph-mon[63029]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-28T12:49:10.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:09 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-28T12:49:10.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:09 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] up:active 2024-04-28T12:49:10.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:09 smithi084 ceph-mon[28081]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 9 up:standby 2024-04-28T12:49:10.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:09 smithi084 ceph-mon[28081]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T12:49:10.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:09 smithi084 ceph-mon[28081]: from='client.24842 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:10.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:09 smithi084 ceph-mon[28081]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-28T12:49:10.196 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.199+0000 7f6a31600640 1 -- 172.21.15.84:0/720491117 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f6a2c062d70 con 0x7f6a100a3c30 2024-04-28T12:49:10.208 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.212+0000 7f6a28c00640 1 -- 172.21.15.84:0/720491117 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f6a2c062d70 con 0x7f6a100a3c30 2024-04-28T12:49:10.208 INFO:teuthology.orchestra.run.smithi084.stdout:/volumes/qa/sv_0/f7256f4e-b207-47b3-bd77-8f559e20cc18 2024-04-28T12:49:10.211 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.215+0000 7f6a31600640 1 -- 172.21.15.84:0/720491117 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a100a3c30 msgr2=0x7f6a100a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:10.211 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.215+0000 7f6a31600640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a100a3c30 0x7f6a100a60f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c19bb80 tx=0x7f6a1c009260 comp rx=0 tx=0).stop 2024-04-28T12:49:10.211 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.215+0000 7f6a31600640 1 -- 172.21.15.84:0/720491117 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a2c105c60 msgr2=0x7f6a2c19b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:10.211 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.215+0000 7f6a31600640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a2c105c60 0x7f6a2c19b1f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6a180057c0 tx=0x7f6a18032da0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.211 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.215+0000 7f6a31600640 1 -- 172.21.15.84:0/720491117 shutdown_connections 2024-04-28T12:49:10.211 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.215+0000 7f6a31600640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a100a3c30 0x7f6a100a60f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.211 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.215+0000 7f6a31600640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a2c106620 0x7f6a2c1a2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.211 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.215+0000 7f6a31600640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a2c105c60 0x7f6a2c19b1f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.211 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.215+0000 7f6a31600640 1 --2- 172.21.15.84:0/720491117 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a2c104a70 0x7f6a2c19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.211 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.215+0000 7f6a31600640 1 -- 172.21.15.84:0/720491117 >> 172.21.15.84:0/720491117 conn(0x7f6a2c0fffe0 msgr2=0x7f6a2c109050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:10.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.215+0000 7f6a31600640 1 -- 172.21.15.84:0/720491117 shutdown_connections 2024-04-28T12:49:10.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.215+0000 7f6a31600640 1 -- 172.21.15.84:0/720491117 wait complete. 2024-04-28T12:49:10.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-28T12:49:10.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:09 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-28T12:49:10.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:09 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] up:active 2024-04-28T12:49:10.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:09 smithi161 ceph-mon[62387]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 9 up:standby 2024-04-28T12:49:10.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:09 smithi161 ceph-mon[62387]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T12:49:10.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:09 smithi161 ceph-mon[62387]: from='client.24842 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:10.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:09 smithi161 ceph-mon[62387]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-28T12:49:10.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.443+0000 7f4ba2800640 1 -- 172.21.15.84:0/739733784 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9c101870 msgr2=0x7f4b9c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:10.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.443+0000 7f4ba2800640 1 --2- 172.21.15.84:0/739733784 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9c101870 0x7f4b9c101cf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c009900 tx=0x7f4b8c02a3f0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.444+0000 7f4ba2800640 1 -- 172.21.15.84:0/739733784 shutdown_connections 2024-04-28T12:49:10.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.444+0000 7f4ba2800640 1 --2- 172.21.15.84:0/739733784 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b9c102230 0x7f4b9c102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.444+0000 7f4ba2800640 1 --2- 172.21.15.84:0/739733784 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9c101870 0x7f4b9c101cf0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.444+0000 7f4ba2800640 1 --2- 172.21.15.84:0/739733784 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4b9c100b10 0x7f4b9c100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.444+0000 7f4ba2800640 1 -- 172.21.15.84:0/739733784 >> 172.21.15.84:0/739733784 conn(0x7f4b9c0fc460 msgr2=0x7f4b9c0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:10.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.444+0000 7f4ba2800640 1 -- 172.21.15.84:0/739733784 shutdown_connections 2024-04-28T12:49:10.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.444+0000 7f4ba2800640 1 -- 172.21.15.84:0/739733784 wait complete. 2024-04-28T12:49:10.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba2800640 1 Processor -- start 2024-04-28T12:49:10.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba2800640 1 -- start start 2024-04-28T12:49:10.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba2800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4b9c100b10 0x7f4b9c19ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:10.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba2800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b9c101870 0x7f4b9c19b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:10.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba2800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9c102230 0x7f4b9c1a2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:10.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba2800640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9c1105d0 con 0x7f4b9c102230 2024-04-28T12:49:10.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba2800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9c110450 con 0x7f4b9c100b10 2024-04-28T12:49:10.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba2800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9c110750 con 0x7f4b9c101870 2024-04-28T12:49:10.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba1e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9c102230 0x7f4b9c1a2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:10.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba1e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9c102230 0x7f4b9c1a2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43188/0 (socket says 172.21.15.84:43188) 2024-04-28T12:49:10.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba1e00640 1 -- 172.21.15.84:0/4146909364 learned_addr learned my addr 172.21.15.84:0/4146909364 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba0a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b9c101870 0x7f4b9c19b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.445+0000 7f4ba1400640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4b9c100b10 0x7f4b9c19ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.446+0000 7f4ba1e00640 1 -- 172.21.15.84:0/4146909364 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b9c101870 msgr2=0x7f4b9c19b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.446+0000 7f4ba1e00640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b9c101870 0x7f4b9c19b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.446+0000 7f4ba1e00640 1 -- 172.21.15.84:0/4146909364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4b9c100b10 msgr2=0x7f4b9c19ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.446+0000 7f4ba1e00640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4b9c100b10 0x7f4b9c19ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.446+0000 7f4ba1e00640 1 -- 172.21.15.84:0/4146909364 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9c1a2930 con 0x7f4b9c102230 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.446+0000 7f4ba1e00640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9c102230 0x7f4b9c1a2290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4b9400eb10 tx=0x7f4b9400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.446+0000 7f4b9a000640 1 -- 172.21.15.84:0/4146909364 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b94004b20 con 0x7f4b9c102230 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.446+0000 7f4b9a000640 1 -- 172.21.15.84:0/4146909364 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b94004cc0 con 0x7f4b9c102230 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.446+0000 7f4ba2800640 1 -- 172.21.15.84:0/4146909364 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9c1a2c20 con 0x7f4b9c102230 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.446+0000 7f4b9a000640 1 -- 172.21.15.84:0/4146909364 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b94010670 con 0x7f4b9c102230 2024-04-28T12:49:10.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.447+0000 7f4ba2800640 1 -- 172.21.15.84:0/4146909364 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b9c1a3120 con 0x7f4b9c102230 2024-04-28T12:49:10.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.448+0000 7f4ba2800640 1 -- 172.21.15.84:0/4146909364 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b9c10c620 con 0x7f4b9c102230 2024-04-28T12:49:10.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.450+0000 7f4b9a000640 1 -- 172.21.15.84:0/4146909364 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4b94005020 con 0x7f4b9c102230 2024-04-28T12:49:10.448 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.451+0000 7f4b9a000640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4b840a3bb0 0x7f4b840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:10.448 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.451+0000 7f4b9a000640 1 -- 172.21.15.84:0/4146909364 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4b940d0c90 con 0x7f4b9c102230 2024-04-28T12:49:10.448 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.452+0000 7f4ba1400640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4b840a3bb0 0x7f4b840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:10.448 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.452+0000 7f4ba1400640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4b840a3bb0 0x7f4b840a6070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c19c1f0 tx=0x7f4b90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:10.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.456+0000 7f4b9a000640 1 -- 172.21.15.84:0/4146909364 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4b9409d3d0 con 0x7f4b9c102230 2024-04-28T12:49:10.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.698+0000 7f4ba2800640 1 -- 172.21.15.84:0/4146909364 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4b9c062d70 con 0x7f4b840a3bb0 2024-04-28T12:49:10.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.735+0000 7f4b9a000640 1 -- 172.21.15.84:0/4146909364 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4b9c062d70 con 0x7f4b840a3bb0 2024-04-28T12:49:10.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.738+0000 7f4ba2800640 1 -- 172.21.15.84:0/4146909364 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4b840a3bb0 msgr2=0x7f4b840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:10.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.738+0000 7f4ba2800640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4b840a3bb0 0x7f4b840a6070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c19c1f0 tx=0x7f4b90005f90 comp rx=0 tx=0).stop 2024-04-28T12:49:10.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.738+0000 7f4ba2800640 1 -- 172.21.15.84:0/4146909364 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9c102230 msgr2=0x7f4b9c1a2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:10.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.738+0000 7f4ba2800640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9c102230 0x7f4b9c1a2290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4b9400eb10 tx=0x7f4b9400c2d0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.738+0000 7f4ba2800640 1 -- 172.21.15.84:0/4146909364 shutdown_connections 2024-04-28T12:49:10.735 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.738+0000 7f4ba2800640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4b840a3bb0 0x7f4b840a6070 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.735 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.738+0000 7f4ba2800640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4b9c102230 0x7f4b9c1a2290 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.735 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.738+0000 7f4ba2800640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4b9c101870 0x7f4b9c19b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.735 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.738+0000 7f4ba2800640 1 --2- 172.21.15.84:0/4146909364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4b9c100b10 0x7f4b9c19ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.735 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.738+0000 7f4ba2800640 1 -- 172.21.15.84:0/4146909364 >> 172.21.15.84:0/4146909364 conn(0x7f4b9c0fc460 msgr2=0x7f4b9c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:10.735 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.738+0000 7f4ba2800640 1 -- 172.21.15.84:0/4146909364 shutdown_connections 2024-04-28T12:49:10.735 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.738+0000 7f4ba2800640 1 -- 172.21.15.84:0/4146909364 wait complete. 2024-04-28T12:49:10.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-28T12:49:10.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.956+0000 7f6931000640 1 -- 172.21.15.84:0/3826476744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f692c105da0 msgr2=0x7f692c106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:10.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.956+0000 7f6931000640 1 --2- 172.21.15.84:0/3826476744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f692c105da0 0x7f692c106220 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6918009860 tx=0x7f691802a340 comp rx=0 tx=0).stop 2024-04-28T12:49:10.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.956+0000 7f6931000640 1 -- 172.21.15.84:0/3826476744 shutdown_connections 2024-04-28T12:49:10.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.956+0000 7f6931000640 1 --2- 172.21.15.84:0/3826476744 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f692c106760 0x7f692c10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.956+0000 7f6931000640 1 --2- 172.21.15.84:0/3826476744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f692c105da0 0x7f692c106220 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.956+0000 7f6931000640 1 --2- 172.21.15.84:0/3826476744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f692c104bb0 0x7f692c104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.956+0000 7f6931000640 1 -- 172.21.15.84:0/3826476744 >> 172.21.15.84:0/3826476744 conn(0x7f692c100120 msgr2=0x7f692c102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:10.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.956+0000 7f6931000640 1 -- 172.21.15.84:0/3826476744 shutdown_connections 2024-04-28T12:49:10.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.956+0000 7f6931000640 1 -- 172.21.15.84:0/3826476744 wait complete. 2024-04-28T12:49:10.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.957+0000 7f6931000640 1 Processor -- start 2024-04-28T12:49:10.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.958+0000 7f6931000640 1 -- start start 2024-04-28T12:49:10.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.958+0000 7f6931000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f692c104bb0 0x7f692c19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:10.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.958+0000 7f6931000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f692c105da0 0x7f692c19b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:10.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.958+0000 7f6931000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f692c106760 0x7f692c1a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:10.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.958+0000 7f6931000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f692c1106e0 con 0x7f692c106760 2024-04-28T12:49:10.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.958+0000 7f6931000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f692c110560 con 0x7f692c105da0 2024-04-28T12:49:10.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.958+0000 7f6931000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f692c110860 con 0x7f692c104bb0 2024-04-28T12:49:10.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.958+0000 7f692a800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f692c106760 0x7f692c1a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:10.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.958+0000 7f692a800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f692c106760 0x7f692c1a23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43208/0 (socket says 172.21.15.84:43208) 2024-04-28T12:49:10.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.958+0000 7f692a800640 1 -- 172.21.15.84:0/4197595697 learned_addr learned my addr 172.21.15.84:0/4197595697 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:10.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.958+0000 7f6929e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f692c104bb0 0x7f692c19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:10.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.958+0000 7f6929400640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f692c105da0 0x7f692c19b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:10.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.959+0000 7f692a800640 1 -- 172.21.15.84:0/4197595697 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f692c104bb0 msgr2=0x7f692c19ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:10.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.959+0000 7f692a800640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f692c104bb0 0x7f692c19ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.959+0000 7f692a800640 1 -- 172.21.15.84:0/4197595697 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f692c105da0 msgr2=0x7f692c19b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:10.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.959+0000 7f692a800640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f692c105da0 0x7f692c19b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:10.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.959+0000 7f692a800640 1 -- 172.21.15.84:0/4197595697 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692c1a2a30 con 0x7f692c106760 2024-04-28T12:49:10.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.959+0000 7f692a800640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f692c106760 0x7f692c1a23c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f692400efc0 tx=0x7f692400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:10.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.959+0000 7f6922a00640 1 -- 172.21.15.84:0/4197595697 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6924003970 con 0x7f692c106760 2024-04-28T12:49:10.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.959+0000 7f6922a00640 1 -- 172.21.15.84:0/4197595697 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6924004020 con 0x7f692c106760 2024-04-28T12:49:10.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.959+0000 7f6931000640 1 -- 172.21.15.84:0/4197595697 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692c1a2d20 con 0x7f692c106760 2024-04-28T12:49:10.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.959+0000 7f6922a00640 1 -- 172.21.15.84:0/4197595697 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692400b940 con 0x7f692c106760 2024-04-28T12:49:10.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.960+0000 7f6931000640 1 -- 172.21.15.84:0/4197595697 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f692c1a3190 con 0x7f692c106760 2024-04-28T12:49:10.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.961+0000 7f6931000640 1 -- 172.21.15.84:0/4197595697 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f692c10bf40 con 0x7f692c106760 2024-04-28T12:49:10.959 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.963+0000 7f6922a00640 1 -- 172.21.15.84:0/4197595697 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f692401d020 con 0x7f692c106760 2024-04-28T12:49:10.961 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.965+0000 7f6922a00640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69100a3c40 0x7f69100a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:10.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.965+0000 7f6922a00640 1 -- 172.21.15.84:0/4197595697 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f69240d06d0 con 0x7f692c106760 2024-04-28T12:49:10.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.966+0000 7f6929e00640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69100a3c40 0x7f69100a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:10.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.966+0000 7f6929e00640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69100a3c40 0x7f69100a6100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f692c105c00 tx=0x7f691c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:10.966 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:10.970+0000 7f6922a00640 1 -- 172.21.15.84:0/4197595697 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f692409ccb0 con 0x7f692c106760 2024-04-28T12:49:11.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:10 smithi179 ceph-mon[63029]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:11.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:10 smithi179 ceph-mon[63029]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:11.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:10 smithi084 ceph-mon[28081]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:11.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:10 smithi084 ceph-mon[28081]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:11.201 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.205+0000 7f6931000640 1 -- 172.21.15.84:0/4197595697 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f692c062e00 con 0x7f69100a3c40 2024-04-28T12:49:11.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.216+0000 7f6922a00640 1 -- 172.21.15.84:0/4197595697 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f692c062e00 con 0x7f69100a3c40 2024-04-28T12:49:11.212 INFO:teuthology.orchestra.run.smithi084.stdout:/volumes/qa/sv_1/9aabb1be-843e-42a1-9aa4-4f80b25dd056 2024-04-28T12:49:11.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.219+0000 7f6931000640 1 -- 172.21.15.84:0/4197595697 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69100a3c40 msgr2=0x7f69100a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:11.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.219+0000 7f6931000640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69100a3c40 0x7f69100a6100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f692c105c00 tx=0x7f691c005f90 comp rx=0 tx=0).stop 2024-04-28T12:49:11.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.219+0000 7f6931000640 1 -- 172.21.15.84:0/4197595697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f692c106760 msgr2=0x7f692c1a23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:11.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.219+0000 7f6931000640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f692c106760 0x7f692c1a23c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f692400efc0 tx=0x7f692400c800 comp rx=0 tx=0).stop 2024-04-28T12:49:11.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.219+0000 7f6931000640 1 -- 172.21.15.84:0/4197595697 shutdown_connections 2024-04-28T12:49:11.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.219+0000 7f6931000640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f69100a3c40 0x7f69100a6100 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.219+0000 7f6931000640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f692c106760 0x7f692c1a23c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.219+0000 7f6931000640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f692c105da0 0x7f692c19b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.219+0000 7f6931000640 1 --2- 172.21.15.84:0/4197595697 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f692c104bb0 0x7f692c19ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.219+0000 7f6931000640 1 -- 172.21.15.84:0/4197595697 >> 172.21.15.84:0/4197595697 conn(0x7f692c100120 msgr2=0x7f692c109110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:11.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.219+0000 7f6931000640 1 -- 172.21.15.84:0/4197595697 shutdown_connections 2024-04-28T12:49:11.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.219+0000 7f6931000640 1 -- 172.21.15.84:0/4197595697 wait complete. 2024-04-28T12:49:11.233 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:49:11.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:10 smithi161 ceph-mon[62387]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:11.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:10 smithi161 ceph-mon[62387]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:11.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.435+0000 7f28eb200640 1 -- 172.21.15.84:0/1236425517 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28e4076910 msgr2=0x7f28e4074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:11.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.435+0000 7f28eb200640 1 --2- 172.21.15.84:0/1236425517 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28e4076910 0x7f28e4074d60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f28d8009900 tx=0x7f28d802a480 comp rx=0 tx=0).stop 2024-04-28T12:49:11.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.436+0000 7f28eb200640 1 -- 172.21.15.84:0/1236425517 shutdown_connections 2024-04-28T12:49:11.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.436+0000 7f28eb200640 1 --2- 172.21.15.84:0/1236425517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e410a640 0x7f28e410cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.436+0000 7f28eb200640 1 --2- 172.21.15.84:0/1236425517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e4075330 0x7f28e4075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.436+0000 7f28eb200640 1 --2- 172.21.15.84:0/1236425517 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28e4076910 0x7f28e4074d60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.436+0000 7f28eb200640 1 -- 172.21.15.84:0/1236425517 >> 172.21.15.84:0/1236425517 conn(0x7f28e40fdd40 msgr2=0x7f28e4100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:11.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.436+0000 7f28eb200640 1 -- 172.21.15.84:0/1236425517 shutdown_connections 2024-04-28T12:49:11.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.436+0000 7f28eb200640 1 -- 172.21.15.84:0/1236425517 wait complete. 2024-04-28T12:49:11.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.437+0000 7f28eb200640 1 Processor -- start 2024-04-28T12:49:11.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.437+0000 7f28eb200640 1 -- start start 2024-04-28T12:49:11.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.437+0000 7f28eb200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28e4075330 0x7f28e419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.437+0000 7f28eb200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e4076910 0x7f28e419b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.437+0000 7f28eb200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e410a640 0x7f28e41a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.437+0000 7f28eb200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f28e4110520 con 0x7f28e4075330 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.437+0000 7f28eb200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f28e41103a0 con 0x7f28e410a640 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.437+0000 7f28eb200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f28e41106a0 con 0x7f28e4076910 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.437+0000 7f28e9e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28e4075330 0x7f28e419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.438+0000 7f28e9e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28e4075330 0x7f28e419ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43220/0 (socket says 172.21.15.84:43220) 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.438+0000 7f28e9e00640 1 -- 172.21.15.84:0/1737025869 learned_addr learned my addr 172.21.15.84:0/1737025869 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.438+0000 7f28ea800640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e410a640 0x7f28e41a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.438+0000 7f28e9400640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e4076910 0x7f28e419b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.438+0000 7f28e9e00640 1 -- 172.21.15.84:0/1737025869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e4076910 msgr2=0x7f28e419b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.438+0000 7f28e9e00640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e4076910 0x7f28e419b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.438+0000 7f28e9e00640 1 -- 172.21.15.84:0/1737025869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e410a640 msgr2=0x7f28e41a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.438+0000 7f28e9e00640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e410a640 0x7f28e41a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.438+0000 7f28e9e00640 1 -- 172.21.15.84:0/1737025869 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e41a27a0 con 0x7f28e4075330 2024-04-28T12:49:11.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.438+0000 7f28e9e00640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28e4075330 0x7f28e419ab80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f28d8005ab0 tx=0x7f28d8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:11.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.439+0000 7f28dea00640 1 -- 172.21.15.84:0/1737025869 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d803f070 con 0x7f28e4075330 2024-04-28T12:49:11.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.439+0000 7f28dea00640 1 -- 172.21.15.84:0/1737025869 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28d8031c50 con 0x7f28e4075330 2024-04-28T12:49:11.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.439+0000 7f28eb200640 1 -- 172.21.15.84:0/1737025869 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e41a2a30 con 0x7f28e4075330 2024-04-28T12:49:11.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.439+0000 7f28dea00640 1 -- 172.21.15.84:0/1737025869 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d8035030 con 0x7f28e4075330 2024-04-28T12:49:11.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.439+0000 7f28eb200640 1 -- 172.21.15.84:0/1737025869 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e41a2f20 con 0x7f28e4075330 2024-04-28T12:49:11.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.440+0000 7f28eb200640 1 -- 172.21.15.84:0/1737025869 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e4109400 con 0x7f28e4075330 2024-04-28T12:49:11.438 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.442+0000 7f28dea00640 1 -- 172.21.15.84:0/1737025869 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f28d8042070 con 0x7f28e4075330 2024-04-28T12:49:11.439 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.443+0000 7f28dea00640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f28cc0a3c40 0x7f28cc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:11.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.443+0000 7f28dea00640 1 -- 172.21.15.84:0/1737025869 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f28d80f0f60 con 0x7f28e4075330 2024-04-28T12:49:11.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.443+0000 7f28e9400640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f28cc0a3c40 0x7f28cc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:11.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.444+0000 7f28e9400640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f28cc0a3c40 0x7f28cc0a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f28e419c090 tx=0x7f28d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:11.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.448+0000 7f28dea00640 1 -- 172.21.15.84:0/1737025869 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f28d80bcef0 con 0x7f28e4075330 2024-04-28T12:49:11.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.692+0000 7f28eb200640 1 -- 172.21.15.84:0/1737025869 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28e402dcf0 con 0x7f28e4075330 2024-04-28T12:49:11.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.693+0000 7f28dea00640 1 -- 172.21.15.84:0/1737025869 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11884 (secure 0 0 0) 0x7f28d80c11f0 con 0x7f28e4075330 2024-04-28T12:49:11.690 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:49:11.690 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6841/1196924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":1196924806},{"type":"v1","addr":"172.21.15.84:6841","nonce":1196924806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:34.334879+0000","modified":"2024-04-28T12:49:08.738419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24562,"mds_2":24545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6839/3812720642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":3812720642},{"type":"v1","addr":"172.21.15.161:6839","nonce":3812720642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24545,24562,24602]},"id":1}]} 2024-04-28T12:49:11.691 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 24 2024-04-28T12:49:11.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.696+0000 7f28eb200640 1 -- 172.21.15.84:0/1737025869 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f28cc0a3c40 msgr2=0x7f28cc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:11.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.696+0000 7f28eb200640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f28cc0a3c40 0x7f28cc0a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f28e419c090 tx=0x7f28d4005f90 comp rx=0 tx=0).stop 2024-04-28T12:49:11.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.696+0000 7f28eb200640 1 -- 172.21.15.84:0/1737025869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28e4075330 msgr2=0x7f28e419ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:11.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.696+0000 7f28eb200640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28e4075330 0x7f28e419ab80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f28d8005ab0 tx=0x7f28d8003b90 comp rx=0 tx=0).stop 2024-04-28T12:49:11.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.696+0000 7f28eb200640 1 -- 172.21.15.84:0/1737025869 shutdown_connections 2024-04-28T12:49:11.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.696+0000 7f28eb200640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f28cc0a3c40 0x7f28cc0a6100 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.696+0000 7f28eb200640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f28e410a640 0x7f28e41a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.696+0000 7f28eb200640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e4076910 0x7f28e419b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.696+0000 7f28eb200640 1 --2- 172.21.15.84:0/1737025869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28e4075330 0x7f28e419ab80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.696+0000 7f28eb200640 1 -- 172.21.15.84:0/1737025869 >> 172.21.15.84:0/1737025869 conn(0x7f28e40fdd40 msgr2=0x7f28e41065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:11.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.696+0000 7f28eb200640 1 -- 172.21.15.84:0/1737025869 shutdown_connections 2024-04-28T12:49:11.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.696+0000 7f28eb200640 1 -- 172.21.15.84:0/1737025869 wait complete. 2024-04-28T12:49:11.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:49:11.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.909+0000 7f8768a00640 1 -- 172.21.15.84:0/1952203076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8764109f80 msgr2=0x7f876410a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:11.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.909+0000 7f8768a00640 1 --2- 172.21.15.84:0/1952203076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8764109f80 0x7f876410a400 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8750009900 tx=0x7f875002a410 comp rx=0 tx=0).stop 2024-04-28T12:49:11.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.909+0000 7f8768a00640 1 -- 172.21.15.84:0/1952203076 shutdown_connections 2024-04-28T12:49:11.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.909+0000 7f8768a00640 1 --2- 172.21.15.84:0/1952203076 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f876410a940 0x7f87641111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.909+0000 7f8768a00640 1 --2- 172.21.15.84:0/1952203076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8764109f80 0x7f876410a400 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.909+0000 7f8768a00640 1 --2- 172.21.15.84:0/1952203076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8764108d90 0x7f87641091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.906 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.909+0000 7f8768a00640 1 -- 172.21.15.84:0/1952203076 >> 172.21.15.84:0/1952203076 conn(0x7f8764077d30 msgr2=0x7f876407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:11.906 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.909+0000 7f8768a00640 1 -- 172.21.15.84:0/1952203076 shutdown_connections 2024-04-28T12:49:11.906 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.909+0000 7f8768a00640 1 -- 172.21.15.84:0/1952203076 wait complete. 2024-04-28T12:49:11.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.910+0000 7f8768a00640 1 Processor -- start 2024-04-28T12:49:11.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.911+0000 7f8768a00640 1 -- start start 2024-04-28T12:49:11.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.911+0000 7f8768a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8764108d90 0x7f876419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:11.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.911+0000 7f8768a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8764109f80 0x7f876419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:11.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.911+0000 7f8768a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f876410a940 0x7f87641a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:11.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.911+0000 7f8768a00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f87641148c0 con 0x7f8764109f80 2024-04-28T12:49:11.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.911+0000 7f8768a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8764114740 con 0x7f8764108d90 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.911+0000 7f8768a00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f8764114a40 con 0x7f876410a940 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.911+0000 7f8762a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8764109f80 0x7f876419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.911+0000 7f8763400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8764108d90 0x7f876419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.911+0000 7f8763e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f876410a940 0x7f87641a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.911+0000 7f8762a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8764109f80 0x7f876419f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43238/0 (socket says 172.21.15.84:43238) 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.911+0000 7f8762a00640 1 -- 172.21.15.84:0/3549018296 learned_addr learned my addr 172.21.15.84:0/3549018296 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.912+0000 7f8763e00640 1 -- 172.21.15.84:0/3549018296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8764108d90 msgr2=0x7f876419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.912+0000 7f8763e00640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8764108d90 0x7f876419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.912+0000 7f8763e00640 1 -- 172.21.15.84:0/3549018296 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8764109f80 msgr2=0x7f876419f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.912+0000 7f8763e00640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8764109f80 0x7f876419f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.912+0000 7f8763e00640 1 -- 172.21.15.84:0/3549018296 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87641a6b90 con 0x7f876410a940 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.912+0000 7f8762a00640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8764109f80 0x7f876419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.912+0000 7f8763e00640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f876410a940 0x7f87641a6520 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f875c009fd0 tx=0x7f875c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.912+0000 7f875be00640 1 -- 172.21.15.84:0/3549018296 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c00ce20 con 0x7f876410a940 2024-04-28T12:49:11.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.912+0000 7f875be00640 1 -- 172.21.15.84:0/3549018296 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f875c005020 con 0x7f876410a940 2024-04-28T12:49:11.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.912+0000 7f8768a00640 1 -- 172.21.15.84:0/3549018296 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87641a6e80 con 0x7f876410a940 2024-04-28T12:49:11.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.913+0000 7f875be00640 1 -- 172.21.15.84:0/3549018296 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c010690 con 0x7f876410a940 2024-04-28T12:49:11.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.913+0000 7f8768a00640 1 -- 172.21.15.84:0/3549018296 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87641a7340 con 0x7f876410a940 2024-04-28T12:49:11.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.914+0000 7f8768a00640 1 -- 172.21.15.84:0/3549018296 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8764110130 con 0x7f876410a940 2024-04-28T12:49:11.911 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.916+0000 7f875be00640 1 -- 172.21.15.84:0/3549018296 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f875c0054d0 con 0x7f876410a940 2024-04-28T12:49:11.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.917+0000 7f875be00640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f87480a3cc0 0x7f87480a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:11.912 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.917+0000 7f875be00640 1 -- 172.21.15.84:0/3549018296 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f875c0d0e70 con 0x7f876410a940 2024-04-28T12:49:11.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.917+0000 7f8763400640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f87480a3cc0 0x7f87480a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:11.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.918+0000 7f8763400640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f87480a3cc0 0x7f87480a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8764109de0 tx=0x7f8754005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:11.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:11.921+0000 7f875be00640 1 -- 172.21.15.84:0/3549018296 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f875c09d450 con 0x7f876410a940 2024-04-28T12:49:12.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.124+0000 7f8768a00640 1 -- 172.21.15.84:0/3549018296 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f876402dd20 con 0x7f876410a940 2024-04-28T12:49:12.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.126+0000 7f875be00640 1 -- 172.21.15.84:0/3549018296 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24117 (secure 0 0 0) 0x7f875c09cd50 con 0x7f876410a940 2024-04-28T12:49:12.122 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:49:12.122 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":90,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:48:40.059255+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:49:12.124 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.129+0000 7f8768a00640 1 -- 172.21.15.84:0/3549018296 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f87480a3cc0 msgr2=0x7f87480a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:12.124 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.129+0000 7f8768a00640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f87480a3cc0 0x7f87480a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8764109de0 tx=0x7f8754005f90 comp rx=0 tx=0).stop 2024-04-28T12:49:12.124 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.129+0000 7f8768a00640 1 -- 172.21.15.84:0/3549018296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f876410a940 msgr2=0x7f87641a6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:12.124 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.129+0000 7f8768a00640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f876410a940 0x7f87641a6520 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f875c009fd0 tx=0x7f875c00ef10 comp rx=0 tx=0).stop 2024-04-28T12:49:12.125 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.129+0000 7f8768a00640 1 -- 172.21.15.84:0/3549018296 shutdown_connections 2024-04-28T12:49:12.125 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.129+0000 7f8768a00640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f87480a3cc0 0x7f87480a6180 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.125 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.129+0000 7f8768a00640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f876410a940 0x7f87641a6520 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.125 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.129+0000 7f8768a00640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8764109f80 0x7f876419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.125 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.129+0000 7f8768a00640 1 --2- 172.21.15.84:0/3549018296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8764108d90 0x7f876419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.125 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.129+0000 7f8768a00640 1 -- 172.21.15.84:0/3549018296 >> 172.21.15.84:0/3549018296 conn(0x7f8764077d30 msgr2=0x7f876410d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:12.125 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.129+0000 7f8768a00640 1 -- 172.21.15.84:0/3549018296 shutdown_connections 2024-04-28T12:49:12.125 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.130+0000 7f8768a00640 1 -- 172.21.15.84:0/3549018296 wait complete. 2024-04-28T12:49:12.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:11 smithi084 ceph-mon[28081]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:12.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:11 smithi084 ceph-mon[28081]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-28T12:49:12.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:11 smithi084 ceph-mon[28081]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:12.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:11 smithi084 ceph-mon[28081]: pgmap v341: 129 pgs: 129 active+clean; 591 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-04-28T12:49:12.142 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:11 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1737025869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:49:12.144 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-28T12:49:12.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-28T12:49:12.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:11 smithi179 ceph-mon[63029]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:12.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:11 smithi179 ceph-mon[63029]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-28T12:49:12.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:11 smithi179 ceph-mon[63029]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:12.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:11 smithi179 ceph-mon[63029]: pgmap v341: 129 pgs: 129 active+clean; 591 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-04-28T12:49:12.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:11 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1737025869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:49:12.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:11 smithi161 ceph-mon[62387]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:12.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:11 smithi161 ceph-mon[62387]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-28T12:49:12.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:11 smithi161 ceph-mon[62387]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:49:12.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:11 smithi161 ceph-mon[62387]: pgmap v341: 129 pgs: 129 active+clean; 591 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-04-28T12:49:12.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:11 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1737025869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:49:12.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.352+0000 7fc0ff400640 1 -- 172.21.15.84:0/2211846387 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f8106720 msgr2=0x7fc0f810cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:12.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.352+0000 7fc0ff400640 1 --2- 172.21.15.84:0/2211846387 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f8106720 0x7fc0f810cfb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0009be0 tx=0x7fc0f002a6c0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.352+0000 7fc0ff400640 1 -- 172.21.15.84:0/2211846387 shutdown_connections 2024-04-28T12:49:12.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.352+0000 7fc0ff400640 1 --2- 172.21.15.84:0/2211846387 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f8106720 0x7fc0f810cfb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.352+0000 7fc0ff400640 1 --2- 172.21.15.84:0/2211846387 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0f8105d60 0x7fc0f81061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.352+0000 7fc0ff400640 1 --2- 172.21.15.84:0/2211846387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0f8104b70 0x7fc0f8104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.352+0000 7fc0ff400640 1 -- 172.21.15.84:0/2211846387 >> 172.21.15.84:0/2211846387 conn(0x7fc0f8100120 msgr2=0x7fc0f8102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:12.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.353+0000 7fc0ff400640 1 -- 172.21.15.84:0/2211846387 shutdown_connections 2024-04-28T12:49:12.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.353+0000 7fc0ff400640 1 -- 172.21.15.84:0/2211846387 wait complete. 2024-04-28T12:49:12.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.353+0000 7fc0ff400640 1 Processor -- start 2024-04-28T12:49:12.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.354+0000 7fc0ff400640 1 -- start start 2024-04-28T12:49:12.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.354+0000 7fc0ff400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f8104b70 0x7fc0f819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:12.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.354+0000 7fc0ff400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0f8105d60 0x7fc0f819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:12.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.354+0000 7fc0ff400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0f8106720 0x7fc0f81a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:12.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.354+0000 7fc0ff400640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81106a0 con 0x7fc0f8104b70 2024-04-28T12:49:12.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.354+0000 7fc0ff400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f8110520 con 0x7fc0f8105d60 2024-04-28T12:49:12.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.354+0000 7fc0ff400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f8110820 con 0x7fc0f8106720 2024-04-28T12:49:12.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.354+0000 7fc0fca00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f8104b70 0x7fc0f819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.354+0000 7fc0fca00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f8104b70 0x7fc0f819ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43260/0 (socket says 172.21.15.84:43260) 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.354+0000 7fc0fca00640 1 -- 172.21.15.84:0/1204685329 learned_addr learned my addr 172.21.15.84:0/1204685329 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.354+0000 7fc0fd400640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0f8106720 0x7fc0f81a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.354+0000 7fc0efe00640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0f8105d60 0x7fc0f819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.355+0000 7fc0fca00640 1 -- 172.21.15.84:0/1204685329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0f8106720 msgr2=0x7fc0f81a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.355+0000 7fc0fca00640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0f8106720 0x7fc0f81a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.355+0000 7fc0fca00640 1 -- 172.21.15.84:0/1204685329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0f8105d60 msgr2=0x7fc0f819b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.355+0000 7fc0fca00640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0f8105d60 0x7fc0f819b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.355+0000 7fc0fca00640 1 -- 172.21.15.84:0/1204685329 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f81a2960 con 0x7fc0f8104b70 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.355+0000 7fc0fca00640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f8104b70 0x7fc0f819ad40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800e9a0 tx=0x7fc0e800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.355+0000 7fc0f6000640 1 -- 172.21.15.84:0/1204685329 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e800cd80 con 0x7fc0f8104b70 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.355+0000 7fc0f6000640 1 -- 172.21.15.84:0/1204685329 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0e80108e0 con 0x7fc0f8104b70 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.355+0000 7fc0f6000640 1 -- 172.21.15.84:0/1204685329 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8018960 con 0x7fc0f8104b70 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.355+0000 7fc0ff400640 1 -- 172.21.15.84:0/1204685329 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f81a2c50 con 0x7fc0f8104b70 2024-04-28T12:49:12.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.355+0000 7fc0ff400640 1 -- 172.21.15.84:0/1204685329 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f81a3150 con 0x7fc0f8104b70 2024-04-28T12:49:12.353 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.356+0000 7fc0ff400640 1 -- 172.21.15.84:0/1204685329 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0f810bf10 con 0x7fc0f8104b70 2024-04-28T12:49:12.353 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.358+0000 7fc0f6000640 1 -- 172.21.15.84:0/1204685329 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc0e8010a90 con 0x7fc0f8104b70 2024-04-28T12:49:12.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.360+0000 7fc0f6000640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc0dc0a3cc0 0x7fc0dc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:12.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.360+0000 7fc0f6000640 1 -- 172.21.15.84:0/1204685329 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc0e80d0aa0 con 0x7fc0f8104b70 2024-04-28T12:49:12.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.360+0000 7fc0efe00640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc0dc0a3cc0 0x7fc0dc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:12.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.361+0000 7fc0efe00640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc0dc0a3cc0 0x7fc0dc0a6180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc0f819c250 tx=0x7fc0e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:12.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.364+0000 7fc0f6000640 1 -- 172.21.15.84:0/1204685329 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc0e8054060 con 0x7fc0f8104b70 2024-04-28T12:49:12.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.564+0000 7fc0ff400640 1 -- 172.21.15.84:0/1204685329 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc0f8062d70 con 0x7fc0dc0a3cc0 2024-04-28T12:49:12.577 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.582+0000 7fc0f6000640 1 -- 172.21.15.84:0/1204685329 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340642 (secure 0 0 0) 0x7fc0f8062d70 con 0x7fc0dc0a3cc0 2024-04-28T12:49:12.583 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:49:12.585 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-28T12:49:12.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.590+0000 7fc0ff400640 1 -- 172.21.15.84:0/1204685329 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc0dc0a3cc0 msgr2=0x7fc0dc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:12.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.590+0000 7fc0ff400640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc0dc0a3cc0 0x7fc0dc0a6180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc0f819c250 tx=0x7fc0e0005f90 comp rx=0 tx=0).stop 2024-04-28T12:49:12.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.590+0000 7fc0ff400640 1 -- 172.21.15.84:0/1204685329 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f8104b70 msgr2=0x7fc0f819ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:12.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.590+0000 7fc0ff400640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f8104b70 0x7fc0f819ad40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800e9a0 tx=0x7fc0e800ee60 comp rx=0 tx=0).stop 2024-04-28T12:49:12.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.590+0000 7fc0ff400640 1 -- 172.21.15.84:0/1204685329 shutdown_connections 2024-04-28T12:49:12.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.590+0000 7fc0ff400640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc0dc0a3cc0 0x7fc0dc0a6180 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.590+0000 7fc0ff400640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0f8106720 0x7fc0f81a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.590+0000 7fc0ff400640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0f8105d60 0x7fc0f819b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.590+0000 7fc0ff400640 1 --2- 172.21.15.84:0/1204685329 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f8104b70 0x7fc0f819ad40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.590+0000 7fc0ff400640 1 -- 172.21.15.84:0/1204685329 >> 172.21.15.84:0/1204685329 conn(0x7fc0f8100120 msgr2=0x7fc0f81090e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:12.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.590+0000 7fc0ff400640 1 -- 172.21.15.84:0/1204685329 shutdown_connections 2024-04-28T12:49:12.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.591+0000 7fc0ff400640 1 -- 172.21.15.84:0/1204685329 wait complete. 2024-04-28T12:49:12.606 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-04-28T12:49:11.609236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":605207,"num_objects":55,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":149,"num_read_kb":131,"num_write":300,"num_write_kb":2985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":242,"ondisk_log_size":242,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365156,"kb_used_data":11832,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576668,"statfs":{"total":1151940427776,"available":1151566508032,"internally_reserved":0,"allocated":12115968,"data_stored":7663432,"data_compressed":283887,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10340,"num_objects":31,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":23,"num_read_kb":22,"num_write":71,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006616"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639620+0000","last_change":"2024-04-28T12:48:34.569210+0000","last_active":"2024-04-28T12:48:41.639620+0000","last_peered":"2024-04-28T12:48:41.639620+0000","last_clean":"2024-04-28T12:48:41.639620+0000","last_became_active":"2024-04-28T12:48:34.568912+0000","last_became_peered":"2024-04-28T12:48:34.568912+0000","last_unstale":"2024-04-28T12:48:41.639620+0000","last_undegraded":"2024-04-28T12:48:41.639620+0000","last_fullsized":"2024-04-28T12:48:41.639620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:48:26.513370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640782+0000","last_change":"2024-04-28T12:48:34.577664+0000","last_active":"2024-04-28T12:48:41.640782+0000","last_peered":"2024-04-28T12:48:41.640782+0000","last_clean":"2024-04-28T12:48:41.640782+0000","last_became_active":"2024-04-28T12:48:34.576679+0000","last_became_peered":"2024-04-28T12:48:34.576679+0000","last_unstale":"2024-04-28T12:48:41.640782+0000","last_undegraded":"2024-04-28T12:48:41.640782+0000","last_fullsized":"2024-04-28T12:48:41.640782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:26:44.807081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.641044+0000","last_change":"2024-04-28T12:48:34.573666+0000","last_active":"2024-04-28T12:48:41.641044+0000","last_peered":"2024-04-28T12:48:41.641044+0000","last_clean":"2024-04-28T12:48:41.641044+0000","last_became_active":"2024-04-28T12:48:34.573111+0000","last_became_peered":"2024-04-28T12:48:34.573111+0000","last_unstale":"2024-04-28T12:48:41.641044+0000","last_undegraded":"2024-04-28T12:48:41.641044+0000","last_fullsized":"2024-04-28T12:48:41.641044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:41:13.586589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639397+0000","last_change":"2024-04-28T12:48:34.543140+0000","last_active":"2024-04-28T12:48:41.639397+0000","last_peered":"2024-04-28T12:48:41.639397+0000","last_clean":"2024-04-28T12:48:41.639397+0000","last_became_active":"2024-04-28T12:48:34.542803+0000","last_became_peered":"2024-04-28T12:48:34.542803+0000","last_unstale":"2024-04-28T12:48:41.639397+0000","last_undegraded":"2024-04-28T12:48:41.639397+0000","last_fullsized":"2024-04-28T12:48:41.639397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:46:47.313190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637643+0000","last_change":"2024-04-28T12:48:34.559988+0000","last_active":"2024-04-28T12:48:41.637643+0000","last_peered":"2024-04-28T12:48:41.637643+0000","last_clean":"2024-04-28T12:48:41.637643+0000","last_became_active":"2024-04-28T12:48:34.559657+0000","last_became_peered":"2024-04-28T12:48:34.559657+0000","last_unstale":"2024-04-28T12:48:41.637643+0000","last_undegraded":"2024-04-28T12:48:41.637643+0000","last_fullsized":"2024-04-28T12:48:41.637643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:13:27.610212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638446+0000","last_change":"2024-04-28T12:48:34.553539+0000","last_active":"2024-04-28T12:48:41.638446+0000","last_peered":"2024-04-28T12:48:41.638446+0000","last_clean":"2024-04-28T12:48:41.638446+0000","last_became_active":"2024-04-28T12:48:34.553017+0000","last_became_peered":"2024-04-28T12:48:34.553017+0000","last_unstale":"2024-04-28T12:48:41.638446+0000","last_undegraded":"2024-04-28T12:48:41.638446+0000","last_fullsized":"2024-04-28T12:48:41.638446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:31:14.895894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638328+0000","last_change":"2024-04-28T12:48:34.553305+0000","last_active":"2024-04-28T12:48:41.638328+0000","last_peered":"2024-04-28T12:48:41.638328+0000","last_clean":"2024-04-28T12:48:41.638328+0000","last_became_active":"2024-04-28T12:48:34.552950+0000","last_became_peered":"2024-04-28T12:48:34.552950+0000","last_unstale":"2024-04-28T12:48:41.638328+0000","last_undegraded":"2024-04-28T12:48:41.638328+0000","last_fullsized":"2024-04-28T12:48:41.638328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:01:43.903986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638745+0000","last_change":"2024-04-28T12:48:34.558899+0000","last_active":"2024-04-28T12:48:41.638745+0000","last_peered":"2024-04-28T12:48:41.638745+0000","last_clean":"2024-04-28T12:48:41.638745+0000","last_became_active":"2024-04-28T12:48:34.558507+0000","last_became_peered":"2024-04-28T12:48:34.558507+0000","last_unstale":"2024-04-28T12:48:41.638745+0000","last_undegraded":"2024-04-28T12:48:41.638745+0000","last_fullsized":"2024-04-28T12:48:41.638745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:46:28.059231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576844+0000","last_change":"2024-04-28T12:48:34.545488+0000","last_active":"2024-04-28T12:48:40.576844+0000","last_peered":"2024-04-28T12:48:40.576844+0000","last_clean":"2024-04-28T12:48:40.576844+0000","last_became_active":"2024-04-28T12:48:34.545168+0000","last_became_peered":"2024-04-28T12:48:34.545168+0000","last_unstale":"2024-04-28T12:48:40.576844+0000","last_undegraded":"2024-04-28T12:48:40.576844+0000","last_fullsized":"2024-04-28T12:48:40.576844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:18:02.174760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639732+0000","last_change":"2024-04-28T12:48:34.544213+0000","last_active":"2024-04-28T12:48:41.639732+0000","last_peered":"2024-04-28T12:48:41.639732+0000","last_clean":"2024-04-28T12:48:41.639732+0000","last_became_active":"2024-04-28T12:48:34.543910+0000","last_became_peered":"2024-04-28T12:48:34.543910+0000","last_unstale":"2024-04-28T12:48:41.639732+0000","last_undegraded":"2024-04-28T12:48:41.639732+0000","last_fullsized":"2024-04-28T12:48:41.639732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:09:52.602695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639075+0000","last_change":"2024-04-28T12:48:34.548121+0000","last_active":"2024-04-28T12:48:41.639075+0000","last_peered":"2024-04-28T12:48:41.639075+0000","last_clean":"2024-04-28T12:48:41.639075+0000","last_became_active":"2024-04-28T12:48:34.547822+0000","last_became_peered":"2024-04-28T12:48:34.547822+0000","last_unstale":"2024-04-28T12:48:41.639075+0000","last_undegraded":"2024-04-28T12:48:41.639075+0000","last_fullsized":"2024-04-28T12:48:41.639075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T12:49:39.838959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639271+0000","last_change":"2024-04-28T12:48:34.544146+0000","last_active":"2024-04-28T12:48:41.639271+0000","last_peered":"2024-04-28T12:48:41.639271+0000","last_clean":"2024-04-28T12:48:41.639271+0000","last_became_active":"2024-04-28T12:48:34.543805+0000","last_became_peered":"2024-04-28T12:48:34.543805+0000","last_unstale":"2024-04-28T12:48:41.639271+0000","last_undegraded":"2024-04-28T12:48:41.639271+0000","last_fullsized":"2024-04-28T12:48:41.639271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:06:46.871316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579249+0000","last_change":"2024-04-28T12:48:34.536765+0000","last_active":"2024-04-28T12:48:40.579249+0000","last_peered":"2024-04-28T12:48:40.579249+0000","last_clean":"2024-04-28T12:48:40.579249+0000","last_became_active":"2024-04-28T12:48:34.535398+0000","last_became_peered":"2024-04-28T12:48:34.535398+0000","last_unstale":"2024-04-28T12:48:40.579249+0000","last_undegraded":"2024-04-28T12:48:40.579249+0000","last_fullsized":"2024-04-28T12:48:40.579249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:54:25.100414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639862+0000","last_change":"2024-04-28T12:48:34.573782+0000","last_active":"2024-04-28T12:48:41.639862+0000","last_peered":"2024-04-28T12:48:41.639862+0000","last_clean":"2024-04-28T12:48:41.639862+0000","last_became_active":"2024-04-28T12:48:34.573361+0000","last_became_peered":"2024-04-28T12:48:34.573361+0000","last_unstale":"2024-04-28T12:48:41.639862+0000","last_undegraded":"2024-04-28T12:48:41.639862+0000","last_fullsized":"2024-04-28T12:48:41.639862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:29:02.746038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639745+0000","last_change":"2024-04-28T12:48:34.570810+0000","last_active":"2024-04-28T12:48:41.639745+0000","last_peered":"2024-04-28T12:48:41.639745+0000","last_clean":"2024-04-28T12:48:41.639745+0000","last_became_active":"2024-04-28T12:48:34.570337+0000","last_became_peered":"2024-04-28T12:48:34.570337+0000","last_unstale":"2024-04-28T12:48:41.639745+0000","last_undegraded":"2024-04-28T12:48:41.639745+0000","last_fullsized":"2024-04-28T12:48:41.639745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:41:36.028710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576249+0000","last_change":"2024-04-28T12:48:34.558802+0000","last_active":"2024-04-28T12:48:40.576249+0000","last_peered":"2024-04-28T12:48:40.576249+0000","last_clean":"2024-04-28T12:48:40.576249+0000","last_became_active":"2024-04-28T12:48:34.558425+0000","last_became_peered":"2024-04-28T12:48:34.558425+0000","last_unstale":"2024-04-28T12:48:40.576249+0000","last_undegraded":"2024-04-28T12:48:40.576249+0000","last_fullsized":"2024-04-28T12:48:40.576249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:03:22.681965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639518+0000","last_change":"2024-04-28T12:48:34.569546+0000","last_active":"2024-04-28T12:48:41.639518+0000","last_peered":"2024-04-28T12:48:41.639518+0000","last_clean":"2024-04-28T12:48:41.639518+0000","last_became_active":"2024-04-28T12:48:34.569231+0000","last_became_peered":"2024-04-28T12:48:34.569231+0000","last_unstale":"2024-04-28T12:48:41.639518+0000","last_undegraded":"2024-04-28T12:48:41.639518+0000","last_fullsized":"2024-04-28T12:48:41.639518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:19:24.491204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.524232+0000","last_change":"2024-04-28T12:48:34.555634+0000","last_active":"2024-04-28T12:48:41.524232+0000","last_peered":"2024-04-28T12:48:41.524232+0000","last_clean":"2024-04-28T12:48:41.524232+0000","last_became_active":"2024-04-28T12:48:34.555054+0000","last_became_peered":"2024-04-28T12:48:34.555054+0000","last_unstale":"2024-04-28T12:48:41.524232+0000","last_undegraded":"2024-04-28T12:48:41.524232+0000","last_fullsized":"2024-04-28T12:48:41.524232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:37.493994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639094+0000","last_change":"2024-04-28T12:48:34.568857+0000","last_active":"2024-04-28T12:48:41.639094+0000","last_peered":"2024-04-28T12:48:41.639094+0000","last_clean":"2024-04-28T12:48:41.639094+0000","last_became_active":"2024-04-28T12:48:34.568550+0000","last_became_peered":"2024-04-28T12:48:34.568550+0000","last_unstale":"2024-04-28T12:48:41.639094+0000","last_undegraded":"2024-04-28T12:48:41.639094+0000","last_fullsized":"2024-04-28T12:48:41.639094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:09:40.709446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638885+0000","last_change":"2024-04-28T12:48:39.608021+0000","last_active":"2024-04-28T12:48:41.638885+0000","last_peered":"2024-04-28T12:48:41.638885+0000","last_clean":"2024-04-28T12:48:41.638885+0000","last_became_active":"2024-04-28T12:48:39.607187+0000","last_became_peered":"2024-04-28T12:48:39.607187+0000","last_unstale":"2024-04-28T12:48:41.638885+0000","last_undegraded":"2024-04-28T12:48:41.638885+0000","last_fullsized":"2024-04-28T12:48:41.638885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:31:26.346414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.524409+0000","last_change":"2024-04-28T12:48:33.839179+0000","last_active":"2024-04-28T12:48:41.524409+0000","last_peered":"2024-04-28T12:48:41.524409+0000","last_clean":"2024-04-28T12:48:41.524409+0000","last_became_active":"2024-04-28T12:48:33.838004+0000","last_became_peered":"2024-04-28T12:48:33.838004+0000","last_unstale":"2024-04-28T12:48:41.524409+0000","last_undegraded":"2024-04-28T12:48:41.524409+0000","last_fullsized":"2024-04-28T12:48:41.524409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:32:14.765652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.641375+0000","last_change":"2024-04-28T12:48:34.571840+0000","last_active":"2024-04-28T12:48:41.641375+0000","last_peered":"2024-04-28T12:48:41.641375+0000","last_clean":"2024-04-28T12:48:41.641375+0000","last_became_active":"2024-04-28T12:48:34.568001+0000","last_became_peered":"2024-04-28T12:48:34.568001+0000","last_unstale":"2024-04-28T12:48:41.641375+0000","last_undegraded":"2024-04-28T12:48:41.641375+0000","last_fullsized":"2024-04-28T12:48:41.641375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:15:22.276123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638570+0000","last_change":"2024-04-28T12:48:39.605055+0000","last_active":"2024-04-28T12:48:41.638570+0000","last_peered":"2024-04-28T12:48:41.638570+0000","last_clean":"2024-04-28T12:48:41.638570+0000","last_became_active":"2024-04-28T12:48:39.604566+0000","last_became_peered":"2024-04-28T12:48:39.604566+0000","last_unstale":"2024-04-28T12:48:41.638570+0000","last_undegraded":"2024-04-28T12:48:41.638570+0000","last_fullsized":"2024-04-28T12:48:41.638570+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:48.945467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.641876+0000","last_change":"2024-04-28T12:48:33.841745+0000","last_active":"2024-04-28T12:48:41.641876+0000","last_peered":"2024-04-28T12:48:41.641876+0000","last_clean":"2024-04-28T12:48:41.641876+0000","last_became_active":"2024-04-28T12:48:33.835003+0000","last_became_peered":"2024-04-28T12:48:33.835003+0000","last_unstale":"2024-04-28T12:48:41.641876+0000","last_undegraded":"2024-04-28T12:48:41.641876+0000","last_fullsized":"2024-04-28T12:48:41.641876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:40:14.253322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.641199+0000","last_change":"2024-04-28T12:48:34.577688+0000","last_active":"2024-04-28T12:48:41.641199+0000","last_peered":"2024-04-28T12:48:41.641199+0000","last_clean":"2024-04-28T12:48:41.641199+0000","last_became_active":"2024-04-28T12:48:34.577269+0000","last_became_peered":"2024-04-28T12:48:34.577269+0000","last_unstale":"2024-04-28T12:48:41.641199+0000","last_undegraded":"2024-04-28T12:48:41.641199+0000","last_fullsized":"2024-04-28T12:48:41.641199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:30:13.744015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638617+0000","last_change":"2024-04-28T12:48:39.600708+0000","last_active":"2024-04-28T12:48:41.638617+0000","last_peered":"2024-04-28T12:48:41.638617+0000","last_clean":"2024-04-28T12:48:41.638617+0000","last_became_active":"2024-04-28T12:48:39.600257+0000","last_became_peered":"2024-04-28T12:48:39.600257+0000","last_unstale":"2024-04-28T12:48:41.638617+0000","last_undegraded":"2024-04-28T12:48:41.638617+0000","last_fullsized":"2024-04-28T12:48:41.638617+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:08:02.685278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638387+0000","last_change":"2024-04-28T12:48:33.837824+0000","last_active":"2024-04-28T12:48:41.638387+0000","last_peered":"2024-04-28T12:48:41.638387+0000","last_clean":"2024-04-28T12:48:41.638387+0000","last_became_active":"2024-04-28T12:48:33.837505+0000","last_became_peered":"2024-04-28T12:48:33.837505+0000","last_unstale":"2024-04-28T12:48:41.638387+0000","last_undegraded":"2024-04-28T12:48:41.638387+0000","last_fullsized":"2024-04-28T12:48:41.638387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:21:10.677381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576533+0000","last_change":"2024-04-28T12:48:34.554575+0000","last_active":"2024-04-28T12:48:40.576533+0000","last_peered":"2024-04-28T12:48:40.576533+0000","last_clean":"2024-04-28T12:48:40.576533+0000","last_became_active":"2024-04-28T12:48:34.554214+0000","last_became_peered":"2024-04-28T12:48:34.554214+0000","last_unstale":"2024-04-28T12:48:40.576533+0000","last_undegraded":"2024-04-28T12:48:40.576533+0000","last_fullsized":"2024-04-28T12:48:40.576533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:37:03.369874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638017+0000","last_change":"2024-04-28T12:48:39.604793+0000","last_active":"2024-04-28T12:48:41.638017+0000","last_peered":"2024-04-28T12:48:41.638017+0000","last_clean":"2024-04-28T12:48:41.638017+0000","last_became_active":"2024-04-28T12:48:39.604338+0000","last_became_peered":"2024-04-28T12:48:39.604338+0000","last_unstale":"2024-04-28T12:48:41.638017+0000","last_undegraded":"2024-04-28T12:48:41.638017+0000","last_fullsized":"2024-04-28T12:48:41.638017+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:14:56.225098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637754+0000","last_change":"2024-04-28T12:48:33.839375+0000","last_active":"2024-04-28T12:48:41.637754+0000","last_peered":"2024-04-28T12:48:41.637754+0000","last_clean":"2024-04-28T12:48:41.637754+0000","last_became_active":"2024-04-28T12:48:33.839062+0000","last_became_peered":"2024-04-28T12:48:33.839062+0000","last_unstale":"2024-04-28T12:48:41.637754+0000","last_undegraded":"2024-04-28T12:48:41.637754+0000","last_fullsized":"2024-04-28T12:48:41.637754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:20:06.692743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576546+0000","last_change":"2024-04-28T12:48:34.543341+0000","last_active":"2024-04-28T12:48:40.576546+0000","last_peered":"2024-04-28T12:48:40.576546+0000","last_clean":"2024-04-28T12:48:40.576546+0000","last_became_active":"2024-04-28T12:48:34.543033+0000","last_became_peered":"2024-04-28T12:48:34.543033+0000","last_unstale":"2024-04-28T12:48:40.576546+0000","last_undegraded":"2024-04-28T12:48:40.576546+0000","last_fullsized":"2024-04-28T12:48:40.576546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:41:27.654577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638950+0000","last_change":"2024-04-28T12:48:39.589856+0000","last_active":"2024-04-28T12:48:41.638950+0000","last_peered":"2024-04-28T12:48:41.638950+0000","last_clean":"2024-04-28T12:48:41.638950+0000","last_became_active":"2024-04-28T12:48:39.589148+0000","last_became_peered":"2024-04-28T12:48:39.589148+0000","last_unstale":"2024-04-28T12:48:41.638950+0000","last_undegraded":"2024-04-28T12:48:41.638950+0000","last_fullsized":"2024-04-28T12:48:41.638950+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:34:11.897109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:08.751314+0000","last_change":"2024-04-28T12:48:33.829387+0000","last_active":"2024-04-28T12:49:08.751314+0000","last_peered":"2024-04-28T12:49:08.751314+0000","last_clean":"2024-04-28T12:49:08.751314+0000","last_became_active":"2024-04-28T12:48:33.829011+0000","last_became_peered":"2024-04-28T12:48:33.829011+0000","last_unstale":"2024-04-28T12:49:08.751314+0000","last_undegraded":"2024-04-28T12:49:08.751314+0000","last_fullsized":"2024-04-28T12:49:08.751314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:54:22.096246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637534+0000","last_change":"2024-04-28T12:48:34.554629+0000","last_active":"2024-04-28T12:48:41.637534+0000","last_peered":"2024-04-28T12:48:41.637534+0000","last_clean":"2024-04-28T12:48:41.637534+0000","last_became_active":"2024-04-28T12:48:34.554231+0000","last_became_peered":"2024-04-28T12:48:34.554231+0000","last_unstale":"2024-04-28T12:48:41.637534+0000","last_undegraded":"2024-04-28T12:48:41.637534+0000","last_fullsized":"2024-04-28T12:48:41.637534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:21:53.706120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638245+0000","last_change":"2024-04-28T12:48:39.606416+0000","last_active":"2024-04-28T12:48:41.638245+0000","last_peered":"2024-04-28T12:48:41.638245+0000","last_clean":"2024-04-28T12:48:41.638245+0000","last_became_active":"2024-04-28T12:48:39.605957+0000","last_became_peered":"2024-04-28T12:48:39.605957+0000","last_unstale":"2024-04-28T12:48:41.638245+0000","last_undegraded":"2024-04-28T12:48:41.638245+0000","last_fullsized":"2024-04-28T12:48:41.638245+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:39:16.315446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:07.739900+0000","last_change":"2024-04-28T12:48:33.841119+0000","last_active":"2024-04-28T12:49:07.739900+0000","last_peered":"2024-04-28T12:49:07.739900+0000","last_clean":"2024-04-28T12:49:07.739900+0000","last_became_active":"2024-04-28T12:48:33.840833+0000","last_became_peered":"2024-04-28T12:48:33.840833+0000","last_unstale":"2024-04-28T12:49:07.739900+0000","last_undegraded":"2024-04-28T12:49:07.739900+0000","last_fullsized":"2024-04-28T12:49:07.739900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:49:31.145968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576510+0000","last_change":"2024-04-28T12:48:34.565140+0000","last_active":"2024-04-28T12:48:40.576510+0000","last_peered":"2024-04-28T12:48:40.576510+0000","last_clean":"2024-04-28T12:48:40.576510+0000","last_became_active":"2024-04-28T12:48:34.564692+0000","last_became_peered":"2024-04-28T12:48:34.564692+0000","last_unstale":"2024-04-28T12:48:40.576510+0000","last_undegraded":"2024-04-28T12:48:40.576510+0000","last_fullsized":"2024-04-28T12:48:40.576510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:55:27.440482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639430+0000","last_change":"2024-04-28T12:48:39.611109+0000","last_active":"2024-04-28T12:48:41.639430+0000","last_peered":"2024-04-28T12:48:41.639430+0000","last_clean":"2024-04-28T12:48:41.639430+0000","last_became_active":"2024-04-28T12:48:39.610754+0000","last_became_peered":"2024-04-28T12:48:39.610754+0000","last_unstale":"2024-04-28T12:48:41.639430+0000","last_undegraded":"2024-04-28T12:48:41.639430+0000","last_fullsized":"2024-04-28T12:48:41.639430+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:55:14.610718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:07.742515+0000","last_change":"2024-04-28T12:48:33.838037+0000","last_active":"2024-04-28T12:49:07.742515+0000","last_peered":"2024-04-28T12:49:07.742515+0000","last_clean":"2024-04-28T12:49:07.742515+0000","last_became_active":"2024-04-28T12:48:33.837704+0000","last_became_peered":"2024-04-28T12:48:33.837704+0000","last_unstale":"2024-04-28T12:49:07.742515+0000","last_undegraded":"2024-04-28T12:49:07.742515+0000","last_fullsized":"2024-04-28T12:49:07.742515+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:46:24.595855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638948+0000","last_change":"2024-04-28T12:48:34.558165+0000","last_active":"2024-04-28T12:48:41.638948+0000","last_peered":"2024-04-28T12:48:41.638948+0000","last_clean":"2024-04-28T12:48:41.638948+0000","last_became_active":"2024-04-28T12:48:34.557728+0000","last_became_peered":"2024-04-28T12:48:34.557728+0000","last_unstale":"2024-04-28T12:48:41.638948+0000","last_undegraded":"2024-04-28T12:48:41.638948+0000","last_fullsized":"2024-04-28T12:48:41.638948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:09:00.169634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640679+0000","last_change":"2024-04-28T12:48:39.615515+0000","last_active":"2024-04-28T12:48:41.640679+0000","last_peered":"2024-04-28T12:48:41.640679+0000","last_clean":"2024-04-28T12:48:41.640679+0000","last_became_active":"2024-04-28T12:48:39.614717+0000","last_became_peered":"2024-04-28T12:48:39.614717+0000","last_unstale":"2024-04-28T12:48:41.640679+0000","last_undegraded":"2024-04-28T12:48:41.640679+0000","last_fullsized":"2024-04-28T12:48:41.640679+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:11:06.633399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.575969+0000","last_change":"2024-04-28T12:48:33.832694+0000","last_active":"2024-04-28T12:48:40.575969+0000","last_peered":"2024-04-28T12:48:40.575969+0000","last_clean":"2024-04-28T12:48:40.575969+0000","last_became_active":"2024-04-28T12:48:33.832381+0000","last_became_peered":"2024-04-28T12:48:33.832381+0000","last_unstale":"2024-04-28T12:48:40.575969+0000","last_undegraded":"2024-04-28T12:48:40.575969+0000","last_fullsized":"2024-04-28T12:48:40.575969+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:56:57.511053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.523396+0000","last_change":"2024-04-28T12:48:34.560229+0000","last_active":"2024-04-28T12:48:41.523396+0000","last_peered":"2024-04-28T12:48:41.523396+0000","last_clean":"2024-04-28T12:48:41.523396+0000","last_became_active":"2024-04-28T12:48:34.559926+0000","last_became_peered":"2024-04-28T12:48:34.559926+0000","last_unstale":"2024-04-28T12:48:41.523396+0000","last_undegraded":"2024-04-28T12:48:41.523396+0000","last_fullsized":"2024-04-28T12:48:41.523396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:38:22.476449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638729+0000","last_change":"2024-04-28T12:48:39.604042+0000","last_active":"2024-04-28T12:48:41.638729+0000","last_peered":"2024-04-28T12:48:41.638729+0000","last_clean":"2024-04-28T12:48:41.638729+0000","last_became_active":"2024-04-28T12:48:39.603631+0000","last_became_peered":"2024-04-28T12:48:39.603631+0000","last_unstale":"2024-04-28T12:48:41.638729+0000","last_undegraded":"2024-04-28T12:48:41.638729+0000","last_fullsized":"2024-04-28T12:48:41.638729+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:53:20.073434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.728038+0000","last_change":"2024-04-28T12:48:33.846372+0000","last_active":"2024-04-28T12:49:06.728038+0000","last_peered":"2024-04-28T12:49:06.728038+0000","last_clean":"2024-04-28T12:49:06.728038+0000","last_became_active":"2024-04-28T12:48:33.846138+0000","last_became_peered":"2024-04-28T12:48:33.846138+0000","last_unstale":"2024-04-28T12:49:06.728038+0000","last_undegraded":"2024-04-28T12:49:06.728038+0000","last_fullsized":"2024-04-28T12:49:06.728038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:46:45.889188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576817+0000","last_change":"2024-04-28T12:48:34.554872+0000","last_active":"2024-04-28T12:48:40.576817+0000","last_peered":"2024-04-28T12:48:40.576817+0000","last_clean":"2024-04-28T12:48:40.576817+0000","last_became_active":"2024-04-28T12:48:34.554555+0000","last_became_peered":"2024-04-28T12:48:34.554555+0000","last_unstale":"2024-04-28T12:48:40.576817+0000","last_undegraded":"2024-04-28T12:48:40.576817+0000","last_fullsized":"2024-04-28T12:48:40.576817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:17:51.280686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.523429+0000","last_change":"2024-04-28T12:48:39.603114+0000","last_active":"2024-04-28T12:48:41.523429+0000","last_peered":"2024-04-28T12:48:41.523429+0000","last_clean":"2024-04-28T12:48:41.523429+0000","last_became_active":"2024-04-28T12:48:39.598404+0000","last_became_peered":"2024-04-28T12:48:39.598404+0000","last_unstale":"2024-04-28T12:48:41.523429+0000","last_undegraded":"2024-04-28T12:48:41.523429+0000","last_fullsized":"2024-04-28T12:48:41.523429+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:23.824317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.727506+0000","last_change":"2024-04-28T12:48:33.831626+0000","last_active":"2024-04-28T12:49:06.727506+0000","last_peered":"2024-04-28T12:49:06.727506+0000","last_clean":"2024-04-28T12:49:06.727506+0000","last_became_active":"2024-04-28T12:48:33.831294+0000","last_became_peered":"2024-04-28T12:48:33.831294+0000","last_unstale":"2024-04-28T12:49:06.727506+0000","last_undegraded":"2024-04-28T12:49:06.727506+0000","last_fullsized":"2024-04-28T12:49:06.727506+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:58:43.775695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640929+0000","last_change":"2024-04-28T12:48:34.574987+0000","last_active":"2024-04-28T12:48:41.640929+0000","last_peered":"2024-04-28T12:48:41.640929+0000","last_clean":"2024-04-28T12:48:41.640929+0000","last_became_active":"2024-04-28T12:48:34.574668+0000","last_became_peered":"2024-04-28T12:48:34.574668+0000","last_unstale":"2024-04-28T12:48:41.640929+0000","last_undegraded":"2024-04-28T12:48:41.640929+0000","last_fullsized":"2024-04-28T12:48:41.640929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:24:06.195671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638944+0000","last_change":"2024-04-28T12:48:39.600246+0000","last_active":"2024-04-28T12:48:41.638944+0000","last_peered":"2024-04-28T12:48:41.638944+0000","last_clean":"2024-04-28T12:48:41.638944+0000","last_became_active":"2024-04-28T12:48:39.599811+0000","last_became_peered":"2024-04-28T12:48:39.599811+0000","last_unstale":"2024-04-28T12:48:41.638944+0000","last_undegraded":"2024-04-28T12:48:41.638944+0000","last_fullsized":"2024-04-28T12:48:41.638944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:32:54.888851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.728380+0000","last_change":"2024-04-28T12:48:33.840884+0000","last_active":"2024-04-28T12:49:06.728380+0000","last_peered":"2024-04-28T12:49:06.728380+0000","last_clean":"2024-04-28T12:49:06.728380+0000","last_became_active":"2024-04-28T12:48:33.840467+0000","last_became_peered":"2024-04-28T12:48:33.840467+0000","last_unstale":"2024-04-28T12:49:06.728380+0000","last_undegraded":"2024-04-28T12:49:06.728380+0000","last_fullsized":"2024-04-28T12:49:06.728380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:28:16.480520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576667+0000","last_change":"2024-04-28T12:48:34.545000+0000","last_active":"2024-04-28T12:48:40.576667+0000","last_peered":"2024-04-28T12:48:40.576667+0000","last_clean":"2024-04-28T12:48:40.576667+0000","last_became_active":"2024-04-28T12:48:34.544662+0000","last_became_peered":"2024-04-28T12:48:34.544662+0000","last_unstale":"2024-04-28T12:48:40.576667+0000","last_undegraded":"2024-04-28T12:48:40.576667+0000","last_fullsized":"2024-04-28T12:48:40.576667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:19:21.640362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.523534+0000","last_change":"2024-04-28T12:48:39.596755+0000","last_active":"2024-04-28T12:48:41.523534+0000","last_peered":"2024-04-28T12:48:41.523534+0000","last_clean":"2024-04-28T12:48:41.523534+0000","last_became_active":"2024-04-28T12:48:39.596292+0000","last_became_peered":"2024-04-28T12:48:39.596292+0000","last_unstale":"2024-04-28T12:48:41.523534+0000","last_undegraded":"2024-04-28T12:48:41.523534+0000","last_fullsized":"2024-04-28T12:48:41.523534+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:21:27.026940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'16","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:09.241584+0000","last_change":"2024-04-28T12:48:33.833840+0000","last_active":"2024-04-28T12:49:09.241584+0000","last_peered":"2024-04-28T12:49:09.241584+0000","last_clean":"2024-04-28T12:49:09.241584+0000","last_became_active":"2024-04-28T12:48:33.833536+0000","last_became_peered":"2024-04-28T12:48:33.833536+0000","last_unstale":"2024-04-28T12:49:09.241584+0000","last_undegraded":"2024-04-28T12:49:09.241584+0000","last_fullsized":"2024-04-28T12:49:09.241584+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:37:01.455373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10442,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638084+0000","last_change":"2024-04-28T12:48:34.548046+0000","last_active":"2024-04-28T12:48:41.638084+0000","last_peered":"2024-04-28T12:48:41.638084+0000","last_clean":"2024-04-28T12:48:41.638084+0000","last_became_active":"2024-04-28T12:48:34.547724+0000","last_became_peered":"2024-04-28T12:48:34.547724+0000","last_unstale":"2024-04-28T12:48:41.638084+0000","last_undegraded":"2024-04-28T12:48:41.638084+0000","last_fullsized":"2024-04-28T12:48:41.638084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:07.287597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576941+0000","last_change":"2024-04-28T12:48:39.602976+0000","last_active":"2024-04-28T12:48:40.576941+0000","last_peered":"2024-04-28T12:48:40.576941+0000","last_clean":"2024-04-28T12:48:40.576941+0000","last_became_active":"2024-04-28T12:48:39.602588+0000","last_became_peered":"2024-04-28T12:48:39.602588+0000","last_unstale":"2024-04-28T12:48:40.576941+0000","last_undegraded":"2024-04-28T12:48:40.576941+0000","last_fullsized":"2024-04-28T12:48:40.576941+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:18:19.549434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:07.741715+0000","last_change":"2024-04-28T12:48:33.831276+0000","last_active":"2024-04-28T12:49:07.741715+0000","last_peered":"2024-04-28T12:49:07.741715+0000","last_clean":"2024-04-28T12:49:07.741715+0000","last_became_active":"2024-04-28T12:48:33.830845+0000","last_became_peered":"2024-04-28T12:48:33.830845+0000","last_unstale":"2024-04-28T12:49:07.741715+0000","last_undegraded":"2024-04-28T12:49:07.741715+0000","last_fullsized":"2024-04-28T12:49:07.741715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:21:02.619309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.578481+0000","last_change":"2024-04-28T12:48:34.566767+0000","last_active":"2024-04-28T12:48:40.578481+0000","last_peered":"2024-04-28T12:48:40.578481+0000","last_clean":"2024-04-28T12:48:40.578481+0000","last_became_active":"2024-04-28T12:48:34.566404+0000","last_became_peered":"2024-04-28T12:48:34.566404+0000","last_unstale":"2024-04-28T12:48:40.578481+0000","last_undegraded":"2024-04-28T12:48:40.578481+0000","last_fullsized":"2024-04-28T12:48:40.578481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:49:32.684046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639425+0000","last_change":"2024-04-28T12:48:39.607471+0000","last_active":"2024-04-28T12:48:41.639425+0000","last_peered":"2024-04-28T12:48:41.639425+0000","last_clean":"2024-04-28T12:48:41.639425+0000","last_became_active":"2024-04-28T12:48:39.607040+0000","last_became_peered":"2024-04-28T12:48:39.607040+0000","last_unstale":"2024-04-28T12:48:41.639425+0000","last_undegraded":"2024-04-28T12:48:41.639425+0000","last_fullsized":"2024-04-28T12:48:41.639425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:06:32.613106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.729004+0000","last_change":"2024-04-28T12:48:33.832519+0000","last_active":"2024-04-28T12:49:06.729004+0000","last_peered":"2024-04-28T12:49:06.729004+0000","last_clean":"2024-04-28T12:49:06.729004+0000","last_became_active":"2024-04-28T12:48:33.832146+0000","last_became_peered":"2024-04-28T12:48:33.832146+0000","last_unstale":"2024-04-28T12:49:06.729004+0000","last_undegraded":"2024-04-28T12:49:06.729004+0000","last_fullsized":"2024-04-28T12:49:06.729004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:02:48.755990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638522+0000","last_change":"2024-04-28T12:48:34.557093+0000","last_active":"2024-04-28T12:48:41.638522+0000","last_peered":"2024-04-28T12:48:41.638522+0000","last_clean":"2024-04-28T12:48:41.638522+0000","last_became_active":"2024-04-28T12:48:34.556767+0000","last_became_peered":"2024-04-28T12:48:34.556767+0000","last_unstale":"2024-04-28T12:48:41.638522+0000","last_undegraded":"2024-04-28T12:48:41.638522+0000","last_fullsized":"2024-04-28T12:48:41.638522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:45:48.634998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.524332+0000","last_change":"2024-04-28T12:48:39.612227+0000","last_active":"2024-04-28T12:48:41.524332+0000","last_peered":"2024-04-28T12:48:41.524332+0000","last_clean":"2024-04-28T12:48:41.524332+0000","last_became_active":"2024-04-28T12:48:39.611065+0000","last_became_peered":"2024-04-28T12:48:39.611065+0000","last_unstale":"2024-04-28T12:48:41.524332+0000","last_undegraded":"2024-04-28T12:48:41.524332+0000","last_fullsized":"2024-04-28T12:48:41.524332+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:15:55.833369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:09.752917+0000","last_change":"2024-04-28T12:48:33.835102+0000","last_active":"2024-04-28T12:49:09.752917+0000","last_peered":"2024-04-28T12:49:09.752917+0000","last_clean":"2024-04-28T12:49:09.752917+0000","last_became_active":"2024-04-28T12:48:33.834672+0000","last_became_peered":"2024-04-28T12:48:33.834672+0000","last_unstale":"2024-04-28T12:49:09.752917+0000","last_undegraded":"2024-04-28T12:49:09.752917+0000","last_fullsized":"2024-04-28T12:49:09.752917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:41:39.388880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637994+0000","last_change":"2024-04-28T12:48:34.548845+0000","last_active":"2024-04-28T12:48:41.637994+0000","last_peered":"2024-04-28T12:48:41.637994+0000","last_clean":"2024-04-28T12:48:41.637994+0000","last_became_active":"2024-04-28T12:48:34.548546+0000","last_became_peered":"2024-04-28T12:48:34.548546+0000","last_unstale":"2024-04-28T12:48:41.637994+0000","last_undegraded":"2024-04-28T12:48:41.637994+0000","last_fullsized":"2024-04-28T12:48:41.637994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:00:53.848671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579467+0000","last_change":"2024-04-28T12:48:39.613039+0000","last_active":"2024-04-28T12:48:40.579467+0000","last_peered":"2024-04-28T12:48:40.579467+0000","last_clean":"2024-04-28T12:48:40.579467+0000","last_became_active":"2024-04-28T12:48:39.609383+0000","last_became_peered":"2024-04-28T12:48:39.609383+0000","last_unstale":"2024-04-28T12:48:40.579467+0000","last_undegraded":"2024-04-28T12:48:40.579467+0000","last_fullsized":"2024-04-28T12:48:40.579467+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:55:47.031973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'10","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.728543+0000","last_change":"2024-04-28T12:48:33.843087+0000","last_active":"2024-04-28T12:49:06.728543+0000","last_peered":"2024-04-28T12:49:06.728543+0000","last_clean":"2024-04-28T12:49:06.728543+0000","last_became_active":"2024-04-28T12:48:33.842775+0000","last_became_peered":"2024-04-28T12:48:33.842775+0000","last_unstale":"2024-04-28T12:49:06.728543+0000","last_undegraded":"2024-04-28T12:49:06.728543+0000","last_fullsized":"2024-04-28T12:49:06.728543+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:39:00.797781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576095+0000","last_change":"2024-04-28T12:48:34.544373+0000","last_active":"2024-04-28T12:48:40.576095+0000","last_peered":"2024-04-28T12:48:40.576095+0000","last_clean":"2024-04-28T12:48:40.576095+0000","last_became_active":"2024-04-28T12:48:34.544040+0000","last_became_peered":"2024-04-28T12:48:34.544040+0000","last_unstale":"2024-04-28T12:48:40.576095+0000","last_undegraded":"2024-04-28T12:48:40.576095+0000","last_fullsized":"2024-04-28T12:48:40.576095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:52:51.088454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576114+0000","last_change":"2024-04-28T12:48:39.608191+0000","last_active":"2024-04-28T12:48:40.576114+0000","last_peered":"2024-04-28T12:48:40.576114+0000","last_clean":"2024-04-28T12:48:40.576114+0000","last_became_active":"2024-04-28T12:48:39.607724+0000","last_became_peered":"2024-04-28T12:48:39.607724+0000","last_unstale":"2024-04-28T12:48:40.576114+0000","last_undegraded":"2024-04-28T12:48:40.576114+0000","last_fullsized":"2024-04-28T12:48:40.576114+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:19:58.040666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638661+0000","last_change":"2024-04-28T12:48:33.839588+0000","last_active":"2024-04-28T12:48:41.638661+0000","last_peered":"2024-04-28T12:48:41.638661+0000","last_clean":"2024-04-28T12:48:41.638661+0000","last_became_active":"2024-04-28T12:48:33.839094+0000","last_became_peered":"2024-04-28T12:48:33.839094+0000","last_unstale":"2024-04-28T12:48:41.638661+0000","last_undegraded":"2024-04-28T12:48:41.638661+0000","last_fullsized":"2024-04-28T12:48:41.638661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:18:35.824206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639495+0000","last_change":"2024-04-28T12:48:34.547354+0000","last_active":"2024-04-28T12:48:41.639495+0000","last_peered":"2024-04-28T12:48:41.639495+0000","last_clean":"2024-04-28T12:48:41.639495+0000","last_became_active":"2024-04-28T12:48:34.546688+0000","last_became_peered":"2024-04-28T12:48:34.546688+0000","last_unstale":"2024-04-28T12:48:41.639495+0000","last_undegraded":"2024-04-28T12:48:41.639495+0000","last_fullsized":"2024-04-28T12:48:41.639495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:10:23.333028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637748+0000","last_change":"2024-04-28T12:48:39.912611+0000","last_active":"2024-04-28T12:48:41.637748+0000","last_peered":"2024-04-28T12:48:41.637748+0000","last_clean":"2024-04-28T12:48:41.637748+0000","last_became_active":"2024-04-28T12:48:39.912097+0000","last_became_peered":"2024-04-28T12:48:39.912097+0000","last_unstale":"2024-04-28T12:48:41.637748+0000","last_undegraded":"2024-04-28T12:48:41.637748+0000","last_fullsized":"2024-04-28T12:48:41.637748+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:31:38.597920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.523912+0000","last_change":"2024-04-28T12:48:33.836343+0000","last_active":"2024-04-28T12:48:41.523912+0000","last_peered":"2024-04-28T12:48:41.523912+0000","last_clean":"2024-04-28T12:48:41.523912+0000","last_became_active":"2024-04-28T12:48:33.835216+0000","last_became_peered":"2024-04-28T12:48:33.835216+0000","last_unstale":"2024-04-28T12:48:41.523912+0000","last_undegraded":"2024-04-28T12:48:41.523912+0000","last_fullsized":"2024-04-28T12:48:41.523912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:44:39.248936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639279+0000","last_change":"2024-04-28T12:48:34.568892+0000","last_active":"2024-04-28T12:48:41.639279+0000","last_peered":"2024-04-28T12:48:41.639279+0000","last_clean":"2024-04-28T12:48:41.639279+0000","last_became_active":"2024-04-28T12:48:34.568581+0000","last_became_peered":"2024-04-28T12:48:34.568581+0000","last_unstale":"2024-04-28T12:48:41.639279+0000","last_undegraded":"2024-04-28T12:48:41.639279+0000","last_fullsized":"2024-04-28T12:48:41.639279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:56:51.661049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579187+0000","last_change":"2024-04-28T12:48:39.616095+0000","last_active":"2024-04-28T12:48:40.579187+0000","last_peered":"2024-04-28T12:48:40.579187+0000","last_clean":"2024-04-28T12:48:40.579187+0000","last_became_active":"2024-04-28T12:48:39.614891+0000","last_became_peered":"2024-04-28T12:48:39.614891+0000","last_unstale":"2024-04-28T12:48:40.579187+0000","last_undegraded":"2024-04-28T12:48:40.579187+0000","last_fullsized":"2024-04-28T12:48:40.579187+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:01:05.645329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638981+0000","last_change":"2024-04-28T12:48:33.833599+0000","last_active":"2024-04-28T12:48:41.638981+0000","last_peered":"2024-04-28T12:48:41.638981+0000","last_clean":"2024-04-28T12:48:41.638981+0000","last_became_active":"2024-04-28T12:48:33.833062+0000","last_became_peered":"2024-04-28T12:48:33.833062+0000","last_unstale":"2024-04-28T12:48:41.638981+0000","last_undegraded":"2024-04-28T12:48:41.638981+0000","last_fullsized":"2024-04-28T12:48:41.638981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:00:13.094007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638529+0000","last_change":"2024-04-28T12:48:34.558759+0000","last_active":"2024-04-28T12:48:41.638529+0000","last_peered":"2024-04-28T12:48:41.638529+0000","last_clean":"2024-04-28T12:48:41.638529+0000","last_became_active":"2024-04-28T12:48:34.558413+0000","last_became_peered":"2024-04-28T12:48:34.558413+0000","last_unstale":"2024-04-28T12:48:41.638529+0000","last_undegraded":"2024-04-28T12:48:41.638529+0000","last_fullsized":"2024-04-28T12:48:41.638529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:07:47.106653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579282+0000","last_change":"2024-04-28T12:48:39.914492+0000","last_active":"2024-04-28T12:48:40.579282+0000","last_peered":"2024-04-28T12:48:40.579282+0000","last_clean":"2024-04-28T12:48:40.579282+0000","last_became_active":"2024-04-28T12:48:39.913113+0000","last_became_peered":"2024-04-28T12:48:39.913113+0000","last_unstale":"2024-04-28T12:48:40.579282+0000","last_undegraded":"2024-04-28T12:48:40.579282+0000","last_fullsized":"2024-04-28T12:48:40.579282+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:49:21.221058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576450+0000","last_change":"2024-04-28T12:48:34.544121+0000","last_active":"2024-04-28T12:48:40.576450+0000","last_peered":"2024-04-28T12:48:40.576450+0000","last_clean":"2024-04-28T12:48:40.576450+0000","last_became_active":"2024-04-28T12:48:34.543816+0000","last_became_peered":"2024-04-28T12:48:34.543816+0000","last_unstale":"2024-04-28T12:48:40.576450+0000","last_undegraded":"2024-04-28T12:48:40.576450+0000","last_fullsized":"2024-04-28T12:48:40.576450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:24.237518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:08.749446+0000","last_change":"2024-04-28T12:48:33.845996+0000","last_active":"2024-04-28T12:49:08.749446+0000","last_peered":"2024-04-28T12:49:08.749446+0000","last_clean":"2024-04-28T12:49:08.749446+0000","last_became_active":"2024-04-28T12:48:33.845618+0000","last_became_peered":"2024-04-28T12:48:33.845618+0000","last_unstale":"2024-04-28T12:49:08.749446+0000","last_undegraded":"2024-04-28T12:49:08.749446+0000","last_fullsized":"2024-04-28T12:49:08.749446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:53:12.312461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638659+0000","last_change":"2024-04-28T12:48:39.589281+0000","last_active":"2024-04-28T12:48:41.638659+0000","last_peered":"2024-04-28T12:48:41.638659+0000","last_clean":"2024-04-28T12:48:41.638659+0000","last_became_active":"2024-04-28T12:48:39.588964+0000","last_became_peered":"2024-04-28T12:48:39.588964+0000","last_unstale":"2024-04-28T12:48:41.638659+0000","last_undegraded":"2024-04-28T12:48:41.638659+0000","last_fullsized":"2024-04-28T12:48:41.638659+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:32:12.039897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639751+0000","last_change":"2024-04-28T12:48:34.567062+0000","last_active":"2024-04-28T12:48:41.639751+0000","last_peered":"2024-04-28T12:48:41.639751+0000","last_clean":"2024-04-28T12:48:41.639751+0000","last_became_active":"2024-04-28T12:48:34.565977+0000","last_became_peered":"2024-04-28T12:48:34.565977+0000","last_unstale":"2024-04-28T12:48:41.639751+0000","last_undegraded":"2024-04-28T12:48:41.639751+0000","last_fullsized":"2024-04-28T12:48:41.639751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:46:03.775693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:08.746870+0000","last_change":"2024-04-28T12:48:33.834069+0000","last_active":"2024-04-28T12:49:08.746870+0000","last_peered":"2024-04-28T12:49:08.746870+0000","last_clean":"2024-04-28T12:49:08.746870+0000","last_became_active":"2024-04-28T12:48:33.833777+0000","last_became_peered":"2024-04-28T12:48:33.833777+0000","last_unstale":"2024-04-28T12:49:08.746870+0000","last_undegraded":"2024-04-28T12:49:08.746870+0000","last_fullsized":"2024-04-28T12:49:08.746870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:32:52.737718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":347,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637848+0000","last_change":"2024-04-28T12:47:13.177589+0000","last_active":"2024-04-28T12:48:41.637848+0000","last_peered":"2024-04-28T12:48:41.637848+0000","last_clean":"2024-04-28T12:48:41.637848+0000","last_became_active":"2024-04-28T12:47:13.177133+0000","last_became_peered":"2024-04-28T12:47:13.177133+0000","last_unstale":"2024-04-28T12:48:41.637848+0000","last_undegraded":"2024-04-28T12:48:41.637848+0000","last_fullsized":"2024-04-28T12:48:41.637848+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:43:15.851674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:43:15.851674+0000","last_clean_scrub_stamp":"2024-04-28T12:43:15.851674+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:23:21.868028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576330+0000","last_change":"2024-04-28T12:48:39.913691+0000","last_active":"2024-04-28T12:48:40.576330+0000","last_peered":"2024-04-28T12:48:40.576330+0000","last_clean":"2024-04-28T12:48:40.576330+0000","last_became_active":"2024-04-28T12:48:39.912989+0000","last_became_peered":"2024-04-28T12:48:39.912989+0000","last_unstale":"2024-04-28T12:48:40.576330+0000","last_undegraded":"2024-04-28T12:48:40.576330+0000","last_fullsized":"2024-04-28T12:48:40.576330+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:49:01.712194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.578039+0000","last_change":"2024-04-28T12:48:34.550976+0000","last_active":"2024-04-28T12:48:40.578039+0000","last_peered":"2024-04-28T12:48:40.578039+0000","last_clean":"2024-04-28T12:48:40.578039+0000","last_became_active":"2024-04-28T12:48:34.549460+0000","last_became_peered":"2024-04-28T12:48:34.549460+0000","last_unstale":"2024-04-28T12:48:40.578039+0000","last_undegraded":"2024-04-28T12:48:40.578039+0000","last_fullsized":"2024-04-28T12:48:40.578039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:55:16.272917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:07.737998+0000","last_change":"2024-04-28T12:48:33.837248+0000","last_active":"2024-04-28T12:49:07.737998+0000","last_peered":"2024-04-28T12:49:07.737998+0000","last_clean":"2024-04-28T12:49:07.737998+0000","last_became_active":"2024-04-28T12:48:33.836910+0000","last_became_peered":"2024-04-28T12:48:33.836910+0000","last_unstale":"2024-04-28T12:49:07.737998+0000","last_undegraded":"2024-04-28T12:49:07.737998+0000","last_fullsized":"2024-04-28T12:49:07.737998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:06:19.572724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639266+0000","last_change":"2024-04-28T12:48:39.605436+0000","last_active":"2024-04-28T12:48:41.639266+0000","last_peered":"2024-04-28T12:48:41.639266+0000","last_clean":"2024-04-28T12:48:41.639266+0000","last_became_active":"2024-04-28T12:48:39.604997+0000","last_became_peered":"2024-04-28T12:48:39.604997+0000","last_unstale":"2024-04-28T12:48:41.639266+0000","last_undegraded":"2024-04-28T12:48:41.639266+0000","last_fullsized":"2024-04-28T12:48:41.639266+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:05:11.571064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638257+0000","last_change":"2024-04-28T12:48:33.848257+0000","last_active":"2024-04-28T12:48:41.638257+0000","last_peered":"2024-04-28T12:48:41.638257+0000","last_clean":"2024-04-28T12:48:41.638257+0000","last_became_active":"2024-04-28T12:48:33.848066+0000","last_became_peered":"2024-04-28T12:48:33.848066+0000","last_unstale":"2024-04-28T12:48:41.638257+0000","last_undegraded":"2024-04-28T12:48:41.638257+0000","last_fullsized":"2024-04-28T12:48:41.638257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:58:34.174153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579714+0000","last_change":"2024-04-28T12:48:34.559791+0000","last_active":"2024-04-28T12:48:40.579714+0000","last_peered":"2024-04-28T12:48:40.579714+0000","last_clean":"2024-04-28T12:48:40.579714+0000","last_became_active":"2024-04-28T12:48:34.556980+0000","last_became_peered":"2024-04-28T12:48:34.556980+0000","last_unstale":"2024-04-28T12:48:40.579714+0000","last_undegraded":"2024-04-28T12:48:40.579714+0000","last_fullsized":"2024-04-28T12:48:40.579714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:38:56.998159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638933+0000","last_change":"2024-04-28T12:48:39.917423+0000","last_active":"2024-04-28T12:48:41.638933+0000","last_peered":"2024-04-28T12:48:41.638933+0000","last_clean":"2024-04-28T12:48:41.638933+0000","last_became_active":"2024-04-28T12:48:39.914893+0000","last_became_peered":"2024-04-28T12:48:39.914893+0000","last_unstale":"2024-04-28T12:48:41.638933+0000","last_undegraded":"2024-04-28T12:48:41.638933+0000","last_fullsized":"2024-04-28T12:48:41.638933+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:08:05.499808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.524270+0000","last_change":"2024-04-28T12:48:33.841383+0000","last_active":"2024-04-28T12:48:41.524270+0000","last_peered":"2024-04-28T12:48:41.524270+0000","last_clean":"2024-04-28T12:48:41.524270+0000","last_became_active":"2024-04-28T12:48:33.838480+0000","last_became_peered":"2024-04-28T12:48:33.838480+0000","last_unstale":"2024-04-28T12:48:41.524270+0000","last_undegraded":"2024-04-28T12:48:41.524270+0000","last_fullsized":"2024-04-28T12:48:41.524270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:53:18.795662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.641395+0000","last_change":"2024-04-28T12:48:34.566120+0000","last_active":"2024-04-28T12:48:41.641395+0000","last_peered":"2024-04-28T12:48:41.641395+0000","last_clean":"2024-04-28T12:48:41.641395+0000","last_became_active":"2024-04-28T12:48:34.564357+0000","last_became_peered":"2024-04-28T12:48:34.564357+0000","last_unstale":"2024-04-28T12:48:41.641395+0000","last_undegraded":"2024-04-28T12:48:41.641395+0000","last_fullsized":"2024-04-28T12:48:41.641395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:28:43.463008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638386+0000","last_change":"2024-04-28T12:48:39.586362+0000","last_active":"2024-04-28T12:48:41.638386+0000","last_peered":"2024-04-28T12:48:41.638386+0000","last_clean":"2024-04-28T12:48:41.638386+0000","last_became_active":"2024-04-28T12:48:39.585898+0000","last_became_peered":"2024-04-28T12:48:39.585898+0000","last_unstale":"2024-04-28T12:48:41.638386+0000","last_undegraded":"2024-04-28T12:48:41.638386+0000","last_fullsized":"2024-04-28T12:48:41.638386+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:51:19.098797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.726144+0000","last_change":"2024-04-28T12:48:33.833163+0000","last_active":"2024-04-28T12:49:06.726144+0000","last_peered":"2024-04-28T12:49:06.726144+0000","last_clean":"2024-04-28T12:49:06.726144+0000","last_became_active":"2024-04-28T12:48:33.832798+0000","last_became_peered":"2024-04-28T12:48:33.832798+0000","last_unstale":"2024-04-28T12:49:06.726144+0000","last_undegraded":"2024-04-28T12:49:06.726144+0000","last_fullsized":"2024-04-28T12:49:06.726144+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:42:07.511346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640879+0000","last_change":"2024-04-28T12:48:34.576917+0000","last_active":"2024-04-28T12:48:41.640879+0000","last_peered":"2024-04-28T12:48:41.640879+0000","last_clean":"2024-04-28T12:48:41.640879+0000","last_became_active":"2024-04-28T12:48:34.575602+0000","last_became_peered":"2024-04-28T12:48:34.575602+0000","last_unstale":"2024-04-28T12:48:41.640879+0000","last_undegraded":"2024-04-28T12:48:41.640879+0000","last_fullsized":"2024-04-28T12:48:41.640879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:47:21.769208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639205+0000","last_change":"2024-04-28T12:48:39.913164+0000","last_active":"2024-04-28T12:48:41.639205+0000","last_peered":"2024-04-28T12:48:41.639205+0000","last_clean":"2024-04-28T12:48:41.639205+0000","last_became_active":"2024-04-28T12:48:39.912716+0000","last_became_peered":"2024-04-28T12:48:39.912716+0000","last_unstale":"2024-04-28T12:48:41.639205+0000","last_undegraded":"2024-04-28T12:48:41.639205+0000","last_fullsized":"2024-04-28T12:48:41.639205+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:26:34.544275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.728395+0000","last_change":"2024-04-28T12:48:33.831286+0000","last_active":"2024-04-28T12:49:06.728395+0000","last_peered":"2024-04-28T12:49:06.728395+0000","last_clean":"2024-04-28T12:49:06.728395+0000","last_became_active":"2024-04-28T12:48:33.830846+0000","last_became_peered":"2024-04-28T12:48:33.830846+0000","last_unstale":"2024-04-28T12:49:06.728395+0000","last_undegraded":"2024-04-28T12:49:06.728395+0000","last_fullsized":"2024-04-28T12:49:06.728395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:32:47.985987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637653+0000","last_change":"2024-04-28T12:48:34.543780+0000","last_active":"2024-04-28T12:48:41.637653+0000","last_peered":"2024-04-28T12:48:41.637653+0000","last_clean":"2024-04-28T12:48:41.637653+0000","last_became_active":"2024-04-28T12:48:34.543441+0000","last_became_peered":"2024-04-28T12:48:34.543441+0000","last_unstale":"2024-04-28T12:48:41.637653+0000","last_undegraded":"2024-04-28T12:48:41.637653+0000","last_fullsized":"2024-04-28T12:48:41.637653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:05:43.334523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.524440+0000","last_change":"2024-04-28T12:48:39.607360+0000","last_active":"2024-04-28T12:48:41.524440+0000","last_peered":"2024-04-28T12:48:41.524440+0000","last_clean":"2024-04-28T12:48:41.524440+0000","last_became_active":"2024-04-28T12:48:39.606944+0000","last_became_peered":"2024-04-28T12:48:39.606944+0000","last_unstale":"2024-04-28T12:48:41.524440+0000","last_undegraded":"2024-04-28T12:48:41.524440+0000","last_fullsized":"2024-04-28T12:48:41.524440+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:59:15.020193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.729166+0000","last_change":"2024-04-28T12:48:33.835694+0000","last_active":"2024-04-28T12:49:06.729166+0000","last_peered":"2024-04-28T12:49:06.729166+0000","last_clean":"2024-04-28T12:49:06.729166+0000","last_became_active":"2024-04-28T12:48:33.835210+0000","last_became_peered":"2024-04-28T12:48:33.835210+0000","last_unstale":"2024-04-28T12:49:06.729166+0000","last_undegraded":"2024-04-28T12:49:06.729166+0000","last_fullsized":"2024-04-28T12:49:06.729166+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:22:34.657665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638809+0000","last_change":"2024-04-28T12:48:34.556933+0000","last_active":"2024-04-28T12:48:41.638809+0000","last_peered":"2024-04-28T12:48:41.638809+0000","last_clean":"2024-04-28T12:48:41.638809+0000","last_became_active":"2024-04-28T12:48:34.556628+0000","last_became_peered":"2024-04-28T12:48:34.556628+0000","last_unstale":"2024-04-28T12:48:41.638809+0000","last_undegraded":"2024-04-28T12:48:41.638809+0000","last_fullsized":"2024-04-28T12:48:41.638809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:40:50.464723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638829+0000","last_change":"2024-04-28T12:48:39.594808+0000","last_active":"2024-04-28T12:48:41.638829+0000","last_peered":"2024-04-28T12:48:41.638829+0000","last_clean":"2024-04-28T12:48:41.638829+0000","last_became_active":"2024-04-28T12:48:39.594333+0000","last_became_peered":"2024-04-28T12:48:39.594333+0000","last_unstale":"2024-04-28T12:48:41.638829+0000","last_undegraded":"2024-04-28T12:48:41.638829+0000","last_fullsized":"2024-04-28T12:48:41.638829+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:34:51.548490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638142+0000","last_change":"2024-04-28T12:48:33.836878+0000","last_active":"2024-04-28T12:48:41.638142+0000","last_peered":"2024-04-28T12:48:41.638142+0000","last_clean":"2024-04-28T12:48:41.638142+0000","last_became_active":"2024-04-28T12:48:33.836536+0000","last_became_peered":"2024-04-28T12:48:33.836536+0000","last_unstale":"2024-04-28T12:48:41.638142+0000","last_undegraded":"2024-04-28T12:48:41.638142+0000","last_fullsized":"2024-04-28T12:48:41.638142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:50:41.064854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640549+0000","last_change":"2024-04-28T12:48:34.575213+0000","last_active":"2024-04-28T12:48:41.640549+0000","last_peered":"2024-04-28T12:48:41.640549+0000","last_clean":"2024-04-28T12:48:41.640549+0000","last_became_active":"2024-04-28T12:48:34.574892+0000","last_became_peered":"2024-04-28T12:48:34.574892+0000","last_unstale":"2024-04-28T12:48:41.640549+0000","last_undegraded":"2024-04-28T12:48:41.640549+0000","last_fullsized":"2024-04-28T12:48:41.640549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:27:54.144215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639384+0000","last_change":"2024-04-28T12:48:39.912466+0000","last_active":"2024-04-28T12:48:41.639384+0000","last_peered":"2024-04-28T12:48:41.639384+0000","last_clean":"2024-04-28T12:48:41.639384+0000","last_became_active":"2024-04-28T12:48:39.911914+0000","last_became_peered":"2024-04-28T12:48:39.911914+0000","last_unstale":"2024-04-28T12:48:41.639384+0000","last_undegraded":"2024-04-28T12:48:41.639384+0000","last_fullsized":"2024-04-28T12:48:41.639384+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:26:35.341389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.726982+0000","last_change":"2024-04-28T12:48:33.840024+0000","last_active":"2024-04-28T12:49:06.726982+0000","last_peered":"2024-04-28T12:49:06.726982+0000","last_clean":"2024-04-28T12:49:06.726982+0000","last_became_active":"2024-04-28T12:48:33.838608+0000","last_became_peered":"2024-04-28T12:48:33.838608+0000","last_unstale":"2024-04-28T12:49:06.726982+0000","last_undegraded":"2024-04-28T12:49:06.726982+0000","last_fullsized":"2024-04-28T12:49:06.726982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:31:28.060385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638864+0000","last_change":"2024-04-28T12:48:34.554685+0000","last_active":"2024-04-28T12:48:41.638864+0000","last_peered":"2024-04-28T12:48:41.638864+0000","last_clean":"2024-04-28T12:48:41.638864+0000","last_became_active":"2024-04-28T12:48:34.554305+0000","last_became_peered":"2024-04-28T12:48:34.554305+0000","last_unstale":"2024-04-28T12:48:41.638864+0000","last_undegraded":"2024-04-28T12:48:41.638864+0000","last_fullsized":"2024-04-28T12:48:41.638864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:19:53.698623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576384+0000","last_change":"2024-04-28T12:48:39.605409+0000","last_active":"2024-04-28T12:48:40.576384+0000","last_peered":"2024-04-28T12:48:40.576384+0000","last_clean":"2024-04-28T12:48:40.576384+0000","last_became_active":"2024-04-28T12:48:39.604991+0000","last_became_peered":"2024-04-28T12:48:39.604991+0000","last_unstale":"2024-04-28T12:48:40.576384+0000","last_undegraded":"2024-04-28T12:48:40.576384+0000","last_fullsized":"2024-04-28T12:48:40.576384+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:37:44.481211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:07.741115+0000","last_change":"2024-04-28T12:48:33.835286+0000","last_active":"2024-04-28T12:49:07.741115+0000","last_peered":"2024-04-28T12:49:07.741115+0000","last_clean":"2024-04-28T12:49:07.741115+0000","last_became_active":"2024-04-28T12:48:33.834729+0000","last_became_peered":"2024-04-28T12:48:33.834729+0000","last_unstale":"2024-04-28T12:49:07.741115+0000","last_undegraded":"2024-04-28T12:49:07.741115+0000","last_fullsized":"2024-04-28T12:49:07.741115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:52:54.889021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639197+0000","last_change":"2024-04-28T12:48:34.547865+0000","last_active":"2024-04-28T12:48:41.639197+0000","last_peered":"2024-04-28T12:48:41.639197+0000","last_clean":"2024-04-28T12:48:41.639197+0000","last_became_active":"2024-04-28T12:48:34.547226+0000","last_became_peered":"2024-04-28T12:48:34.547226+0000","last_unstale":"2024-04-28T12:48:41.639197+0000","last_undegraded":"2024-04-28T12:48:41.639197+0000","last_fullsized":"2024-04-28T12:48:41.639197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:14:06.376908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640100+0000","last_change":"2024-04-28T12:48:39.611010+0000","last_active":"2024-04-28T12:48:41.640100+0000","last_peered":"2024-04-28T12:48:41.640100+0000","last_clean":"2024-04-28T12:48:41.640100+0000","last_became_active":"2024-04-28T12:48:39.610475+0000","last_became_peered":"2024-04-28T12:48:39.610475+0000","last_unstale":"2024-04-28T12:48:41.640100+0000","last_undegraded":"2024-04-28T12:48:41.640100+0000","last_fullsized":"2024-04-28T12:48:41.640100+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:19:44.707837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576717+0000","last_change":"2024-04-28T12:48:33.835109+0000","last_active":"2024-04-28T12:48:40.576717+0000","last_peered":"2024-04-28T12:48:40.576717+0000","last_clean":"2024-04-28T12:48:40.576717+0000","last_became_active":"2024-04-28T12:48:33.834757+0000","last_became_peered":"2024-04-28T12:48:33.834757+0000","last_unstale":"2024-04-28T12:48:40.576717+0000","last_undegraded":"2024-04-28T12:48:40.576717+0000","last_fullsized":"2024-04-28T12:48:40.576717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:46:45.884261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638235+0000","last_change":"2024-04-28T12:48:34.565135+0000","last_active":"2024-04-28T12:48:41.638235+0000","last_peered":"2024-04-28T12:48:41.638235+0000","last_clean":"2024-04-28T12:48:41.638235+0000","last_became_active":"2024-04-28T12:48:34.564631+0000","last_became_peered":"2024-04-28T12:48:34.564631+0000","last_unstale":"2024-04-28T12:48:41.638235+0000","last_undegraded":"2024-04-28T12:48:41.638235+0000","last_fullsized":"2024-04-28T12:48:41.638235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:36:46.747085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.575869+0000","last_change":"2024-04-28T12:48:39.591530+0000","last_active":"2024-04-28T12:48:40.575869+0000","last_peered":"2024-04-28T12:48:40.575869+0000","last_clean":"2024-04-28T12:48:40.575869+0000","last_became_active":"2024-04-28T12:48:39.591016+0000","last_became_peered":"2024-04-28T12:48:39.591016+0000","last_unstale":"2024-04-28T12:48:40.575869+0000","last_undegraded":"2024-04-28T12:48:40.575869+0000","last_fullsized":"2024-04-28T12:48:40.575869+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:04:32.281117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579649+0000","last_change":"2024-04-28T12:48:33.835565+0000","last_active":"2024-04-28T12:48:40.579649+0000","last_peered":"2024-04-28T12:48:40.579649+0000","last_clean":"2024-04-28T12:48:40.579649+0000","last_became_active":"2024-04-28T12:48:33.835147+0000","last_became_peered":"2024-04-28T12:48:33.835147+0000","last_unstale":"2024-04-28T12:48:40.579649+0000","last_undegraded":"2024-04-28T12:48:40.579649+0000","last_fullsized":"2024-04-28T12:48:40.579649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:48:23.520295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639449+0000","last_change":"2024-04-28T12:48:34.547561+0000","last_active":"2024-04-28T12:48:41.639449+0000","last_peered":"2024-04-28T12:48:41.639449+0000","last_clean":"2024-04-28T12:48:41.639449+0000","last_became_active":"2024-04-28T12:48:34.547236+0000","last_became_peered":"2024-04-28T12:48:34.547236+0000","last_unstale":"2024-04-28T12:48:41.639449+0000","last_undegraded":"2024-04-28T12:48:41.639449+0000","last_fullsized":"2024-04-28T12:48:41.639449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:00:37.150080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639007+0000","last_change":"2024-04-28T12:48:34.546027+0000","last_active":"2024-04-28T12:48:41.639007+0000","last_peered":"2024-04-28T12:48:41.639007+0000","last_clean":"2024-04-28T12:48:41.639007+0000","last_became_active":"2024-04-28T12:48:34.545618+0000","last_became_peered":"2024-04-28T12:48:34.545618+0000","last_unstale":"2024-04-28T12:48:41.639007+0000","last_undegraded":"2024-04-28T12:48:41.639007+0000","last_fullsized":"2024-04-28T12:48:41.639007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:40:00.928586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638680+0000","last_change":"2024-04-28T12:48:34.544883+0000","last_active":"2024-04-28T12:48:41.638680+0000","last_peered":"2024-04-28T12:48:41.638680+0000","last_clean":"2024-04-28T12:48:41.638680+0000","last_became_active":"2024-04-28T12:48:34.544454+0000","last_became_peered":"2024-04-28T12:48:34.544454+0000","last_unstale":"2024-04-28T12:48:41.638680+0000","last_undegraded":"2024-04-28T12:48:41.638680+0000","last_fullsized":"2024-04-28T12:48:41.638680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:58:16.064432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637981+0000","last_change":"2024-04-28T12:48:34.565110+0000","last_active":"2024-04-28T12:48:41.637981+0000","last_peered":"2024-04-28T12:48:41.637981+0000","last_clean":"2024-04-28T12:48:41.637981+0000","last_became_active":"2024-04-28T12:48:34.564723+0000","last_became_peered":"2024-04-28T12:48:34.564723+0000","last_unstale":"2024-04-28T12:48:41.637981+0000","last_undegraded":"2024-04-28T12:48:41.637981+0000","last_fullsized":"2024-04-28T12:48:41.637981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:58:24.220515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640384+0000","last_change":"2024-04-28T12:48:34.572428+0000","last_active":"2024-04-28T12:48:41.640384+0000","last_peered":"2024-04-28T12:48:41.640384+0000","last_clean":"2024-04-28T12:48:41.640384+0000","last_became_active":"2024-04-28T12:48:34.572034+0000","last_became_peered":"2024-04-28T12:48:34.572034+0000","last_unstale":"2024-04-28T12:48:41.640384+0000","last_undegraded":"2024-04-28T12:48:41.640384+0000","last_fullsized":"2024-04-28T12:48:41.640384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:38:23.598568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579659+0000","last_change":"2024-04-28T12:48:34.562425+0000","last_active":"2024-04-28T12:48:40.579659+0000","last_peered":"2024-04-28T12:48:40.579659+0000","last_clean":"2024-04-28T12:48:40.579659+0000","last_became_active":"2024-04-28T12:48:34.562087+0000","last_became_peered":"2024-04-28T12:48:34.562087+0000","last_unstale":"2024-04-28T12:48:40.579659+0000","last_undegraded":"2024-04-28T12:48:40.579659+0000","last_fullsized":"2024-04-28T12:48:40.579659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:32:39.701796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639546+0000","last_change":"2024-04-28T12:48:34.547755+0000","last_active":"2024-04-28T12:48:41.639546+0000","last_peered":"2024-04-28T12:48:41.639546+0000","last_clean":"2024-04-28T12:48:41.639546+0000","last_became_active":"2024-04-28T12:48:34.547427+0000","last_became_peered":"2024-04-28T12:48:34.547427+0000","last_unstale":"2024-04-28T12:48:41.639546+0000","last_undegraded":"2024-04-28T12:48:41.639546+0000","last_fullsized":"2024-04-28T12:48:41.639546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:58:15.694252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638706+0000","last_change":"2024-04-28T12:48:34.543146+0000","last_active":"2024-04-28T12:48:41.638706+0000","last_peered":"2024-04-28T12:48:41.638706+0000","last_clean":"2024-04-28T12:48:41.638706+0000","last_became_active":"2024-04-28T12:48:34.542809+0000","last_became_peered":"2024-04-28T12:48:34.542809+0000","last_unstale":"2024-04-28T12:48:41.638706+0000","last_undegraded":"2024-04-28T12:48:41.638706+0000","last_fullsized":"2024-04-28T12:48:41.638706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:09:42.515358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.641607+0000","last_change":"2024-04-28T12:48:34.571166+0000","last_active":"2024-04-28T12:48:41.641607+0000","last_peered":"2024-04-28T12:48:41.641607+0000","last_clean":"2024-04-28T12:48:41.641607+0000","last_became_active":"2024-04-28T12:48:34.567803+0000","last_became_peered":"2024-04-28T12:48:34.567803+0000","last_unstale":"2024-04-28T12:48:41.641607+0000","last_undegraded":"2024-04-28T12:48:41.641607+0000","last_fullsized":"2024-04-28T12:48:41.641607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:11:09.945751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576353+0000","last_change":"2024-04-28T12:48:34.564939+0000","last_active":"2024-04-28T12:48:40.576353+0000","last_peered":"2024-04-28T12:48:40.576353+0000","last_clean":"2024-04-28T12:48:40.576353+0000","last_became_active":"2024-04-28T12:48:34.564486+0000","last_became_peered":"2024-04-28T12:48:34.564486+0000","last_unstale":"2024-04-28T12:48:40.576353+0000","last_undegraded":"2024-04-28T12:48:40.576353+0000","last_fullsized":"2024-04-28T12:48:40.576353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:29:44.618469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639204+0000","last_change":"2024-04-28T12:48:34.567359+0000","last_active":"2024-04-28T12:48:41.639204+0000","last_peered":"2024-04-28T12:48:41.639204+0000","last_clean":"2024-04-28T12:48:41.639204+0000","last_became_active":"2024-04-28T12:48:34.565393+0000","last_became_peered":"2024-04-28T12:48:34.565393+0000","last_unstale":"2024-04-28T12:48:41.639204+0000","last_undegraded":"2024-04-28T12:48:41.639204+0000","last_fullsized":"2024-04-28T12:48:41.639204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:44:04.312257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576675+0000","last_change":"2024-04-28T12:48:34.565019+0000","last_active":"2024-04-28T12:48:40.576675+0000","last_peered":"2024-04-28T12:48:40.576675+0000","last_clean":"2024-04-28T12:48:40.576675+0000","last_became_active":"2024-04-28T12:48:34.564635+0000","last_became_peered":"2024-04-28T12:48:34.564635+0000","last_unstale":"2024-04-28T12:48:40.576675+0000","last_undegraded":"2024-04-28T12:48:40.576675+0000","last_fullsized":"2024-04-28T12:48:40.576675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:20.645281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579260+0000","last_change":"2024-04-28T12:48:34.574043+0000","last_active":"2024-04-28T12:48:40.579260+0000","last_peered":"2024-04-28T12:48:40.579260+0000","last_clean":"2024-04-28T12:48:40.579260+0000","last_became_active":"2024-04-28T12:48:34.573686+0000","last_became_peered":"2024-04-28T12:48:34.573686+0000","last_unstale":"2024-04-28T12:48:40.579260+0000","last_undegraded":"2024-04-28T12:48:40.579260+0000","last_fullsized":"2024-04-28T12:48:40.579260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:29:57.371318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639294+0000","last_change":"2024-04-28T12:48:34.547404+0000","last_active":"2024-04-28T12:48:41.639294+0000","last_peered":"2024-04-28T12:48:41.639294+0000","last_clean":"2024-04-28T12:48:41.639294+0000","last_became_active":"2024-04-28T12:48:34.546581+0000","last_became_peered":"2024-04-28T12:48:34.546581+0000","last_unstale":"2024-04-28T12:48:41.639294+0000","last_undegraded":"2024-04-28T12:48:41.639294+0000","last_fullsized":"2024-04-28T12:48:41.639294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:43:21.389455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281028,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14722,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":23,"num_read_kb":22,"num_write":118,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":86502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":485394,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645337,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1081182,"data_compressed":50466,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808864,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":486178,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070501,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":485998,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332140,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1086528,"data_compressed":50466,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495665,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":485484,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":525865,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":486872,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":490586,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182467,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":485678,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":486429,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":1077238,"data_compressed":50466,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34101,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34101,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34101,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":40471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":24882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-28T12:49:12.613 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-28T12:49:12.853 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.858+0000 7ff7dae00640 1 -- 172.21.15.84:0/2387084177 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7d40772f0 msgr2=0x7ff7d4077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:12.854 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.858+0000 7ff7dae00640 1 --2- 172.21.15.84:0/2387084177 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7d40772f0 0x7ff7d4077710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009ca0 tx=0x7ff7cc02a8c0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.854 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.858+0000 7ff7dae00640 1 -- 172.21.15.84:0/2387084177 shutdown_connections 2024-04-28T12:49:12.854 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.858+0000 7ff7dae00640 1 --2- 172.21.15.84:0/2387084177 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d4078ed0 0x7ff7d4079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.854 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.858+0000 7ff7dae00640 1 --2- 172.21.15.84:0/2387084177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d4078530 0x7ff7d4078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.854 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.858+0000 7ff7dae00640 1 --2- 172.21.15.84:0/2387084177 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7d40772f0 0x7ff7d4077710 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.854 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.858+0000 7ff7dae00640 1 -- 172.21.15.84:0/2387084177 >> 172.21.15.84:0/2387084177 conn(0x7ff7d40fff40 msgr2=0x7ff7d4102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:12.854 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.858+0000 7ff7dae00640 1 -- 172.21.15.84:0/2387084177 shutdown_connections 2024-04-28T12:49:12.854 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.858+0000 7ff7dae00640 1 -- 172.21.15.84:0/2387084177 wait complete. 2024-04-28T12:49:12.855 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.859+0000 7ff7dae00640 1 Processor -- start 2024-04-28T12:49:12.856 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.859+0000 7ff7dae00640 1 -- start start 2024-04-28T12:49:12.856 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7dae00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7d40772f0 0x7ff7d419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:12.856 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7dae00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d4078530 0x7ff7d419f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:12.856 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7dae00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d4078ed0 0x7ff7d41a6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:12.856 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7dae00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d4114930 con 0x7ff7d40772f0 2024-04-28T12:49:12.856 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7dae00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41147b0 con 0x7ff7d4078530 2024-04-28T12:49:12.856 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7dae00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d4114ab0 con 0x7ff7d4078ed0 2024-04-28T12:49:12.856 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7d9a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7d40772f0 0x7ff7d419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:12.856 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7d9a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7d40772f0 0x7ff7d419ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43280/0 (socket says 172.21.15.84:43280) 2024-04-28T12:49:12.856 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7d9a00640 1 -- 172.21.15.84:0/814998979 learned_addr learned my addr 172.21.15.84:0/814998979 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:12.856 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7d9000640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d4078530 0x7ff7d419f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:12.856 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7da400640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d4078ed0 0x7ff7d41a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:12.857 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7d9a00640 1 -- 172.21.15.84:0/814998979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d4078ed0 msgr2=0x7ff7d41a6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:12.857 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7d9a00640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d4078ed0 0x7ff7d41a6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.857 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7d9a00640 1 -- 172.21.15.84:0/814998979 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d4078530 msgr2=0x7ff7d419f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:12.857 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7d9a00640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d4078530 0x7ff7d419f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:12.857 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.860+0000 7ff7d9a00640 1 -- 172.21.15.84:0/814998979 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d41a6b80 con 0x7ff7d40772f0 2024-04-28T12:49:12.857 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.861+0000 7ff7d9a00640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7d40772f0 0x7ff7d419ef60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc000f80 tx=0x7ff7cc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:12.858 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.861+0000 7ff7d2000640 1 -- 172.21.15.84:0/814998979 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc02f380 con 0x7ff7d40772f0 2024-04-28T12:49:12.858 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.861+0000 7ff7d2000640 1 -- 172.21.15.84:0/814998979 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7cc0317c0 con 0x7ff7d40772f0 2024-04-28T12:49:12.858 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.861+0000 7ff7dae00640 1 -- 172.21.15.84:0/814998979 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d41a6e10 con 0x7ff7d40772f0 2024-04-28T12:49:12.858 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.861+0000 7ff7d2000640 1 -- 172.21.15.84:0/814998979 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc02fbe0 con 0x7ff7d40772f0 2024-04-28T12:49:12.858 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.861+0000 7ff7dae00640 1 -- 172.21.15.84:0/814998979 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d41a7270 con 0x7ff7d40772f0 2024-04-28T12:49:12.858 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.862+0000 7ff7dae00640 1 -- 172.21.15.84:0/814998979 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7d4111320 con 0x7ff7d40772f0 2024-04-28T12:49:12.859 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.864+0000 7ff7d2000640 1 -- 172.21.15.84:0/814998979 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff7cc02fd80 con 0x7ff7d40772f0 2024-04-28T12:49:12.861 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.865+0000 7ff7d2000640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7b80a3c40 0x7ff7b80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:12.861 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.866+0000 7ff7d2000640 1 -- 172.21.15.84:0/814998979 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff7cc0f1bf0 con 0x7ff7d40772f0 2024-04-28T12:49:12.861 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.866+0000 7ff7d9000640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7b80a3c40 0x7ff7b80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:12.862 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.867+0000 7ff7d9000640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7b80a3c40 0x7ff7b80a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff7d41a0470 tx=0x7ff7c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:12.865 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:12.870+0000 7ff7d2000640 1 -- 172.21.15.84:0/814998979 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff7cc0be2e0 con 0x7ff7d40772f0 2024-04-28T12:49:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:12 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3549018296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:12.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:12 smithi084 ceph-mon[28081]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:49:13.065 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.070+0000 7ff7dae00640 1 -- 172.21.15.84:0/814998979 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7d4062d70 con 0x7ff7b80a3c40 2024-04-28T12:49:13.082 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.084+0000 7ff7d2000640 1 -- 172.21.15.84:0/814998979 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340642 (secure 0 0 0) 0x7ff7d4062d70 con 0x7ff7b80a3c40 2024-04-28T12:49:13.086 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:49:13.087 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-28T12:49:13.089 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.092+0000 7ff7dae00640 1 -- 172.21.15.84:0/814998979 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7b80a3c40 msgr2=0x7ff7b80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:13.089 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.092+0000 7ff7dae00640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7b80a3c40 0x7ff7b80a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff7d41a0470 tx=0x7ff7c0005f90 comp rx=0 tx=0).stop 2024-04-28T12:49:13.089 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.092+0000 7ff7dae00640 1 -- 172.21.15.84:0/814998979 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7d40772f0 msgr2=0x7ff7d419ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:13.089 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.092+0000 7ff7dae00640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7d40772f0 0x7ff7d419ef60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc000f80 tx=0x7ff7cc003ba0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.092+0000 7ff7dae00640 1 -- 172.21.15.84:0/814998979 shutdown_connections 2024-04-28T12:49:13.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.092+0000 7ff7dae00640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7b80a3c40 0x7ff7b80a6100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.092+0000 7ff7dae00640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7d4078ed0 0x7ff7d41a6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.092+0000 7ff7dae00640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d4078530 0x7ff7d419f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.092+0000 7ff7dae00640 1 --2- 172.21.15.84:0/814998979 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7d40772f0 0x7ff7d419ef60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:13.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.092+0000 7ff7dae00640 1 -- 172.21.15.84:0/814998979 >> 172.21.15.84:0/814998979 conn(0x7ff7d40fff40 msgr2=0x7ff7d4101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:13.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.093+0000 7ff7dae00640 1 -- 172.21.15.84:0/814998979 shutdown_connections 2024-04-28T12:49:13.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:13.093+0000 7ff7dae00640 1 -- 172.21.15.84:0/814998979 wait complete. 2024-04-28T12:49:13.107 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-04-28T12:49:11.609236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":605207,"num_objects":55,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":149,"num_read_kb":131,"num_write":300,"num_write_kb":2985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":242,"ondisk_log_size":242,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365156,"kb_used_data":11832,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576668,"statfs":{"total":1151940427776,"available":1151566508032,"internally_reserved":0,"allocated":12115968,"data_stored":7663432,"data_compressed":283887,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10340,"num_objects":31,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":23,"num_read_kb":22,"num_write":71,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006616"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639620+0000","last_change":"2024-04-28T12:48:34.569210+0000","last_active":"2024-04-28T12:48:41.639620+0000","last_peered":"2024-04-28T12:48:41.639620+0000","last_clean":"2024-04-28T12:48:41.639620+0000","last_became_active":"2024-04-28T12:48:34.568912+0000","last_became_peered":"2024-04-28T12:48:34.568912+0000","last_unstale":"2024-04-28T12:48:41.639620+0000","last_undegraded":"2024-04-28T12:48:41.639620+0000","last_fullsized":"2024-04-28T12:48:41.639620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:48:26.513370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640782+0000","last_change":"2024-04-28T12:48:34.577664+0000","last_active":"2024-04-28T12:48:41.640782+0000","last_peered":"2024-04-28T12:48:41.640782+0000","last_clean":"2024-04-28T12:48:41.640782+0000","last_became_active":"2024-04-28T12:48:34.576679+0000","last_became_peered":"2024-04-28T12:48:34.576679+0000","last_unstale":"2024-04-28T12:48:41.640782+0000","last_undegraded":"2024-04-28T12:48:41.640782+0000","last_fullsized":"2024-04-28T12:48:41.640782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:26:44.807081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.641044+0000","last_change":"2024-04-28T12:48:34.573666+0000","last_active":"2024-04-28T12:48:41.641044+0000","last_peered":"2024-04-28T12:48:41.641044+0000","last_clean":"2024-04-28T12:48:41.641044+0000","last_became_active":"2024-04-28T12:48:34.573111+0000","last_became_peered":"2024-04-28T12:48:34.573111+0000","last_unstale":"2024-04-28T12:48:41.641044+0000","last_undegraded":"2024-04-28T12:48:41.641044+0000","last_fullsized":"2024-04-28T12:48:41.641044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:41:13.586589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639397+0000","last_change":"2024-04-28T12:48:34.543140+0000","last_active":"2024-04-28T12:48:41.639397+0000","last_peered":"2024-04-28T12:48:41.639397+0000","last_clean":"2024-04-28T12:48:41.639397+0000","last_became_active":"2024-04-28T12:48:34.542803+0000","last_became_peered":"2024-04-28T12:48:34.542803+0000","last_unstale":"2024-04-28T12:48:41.639397+0000","last_undegraded":"2024-04-28T12:48:41.639397+0000","last_fullsized":"2024-04-28T12:48:41.639397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:46:47.313190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637643+0000","last_change":"2024-04-28T12:48:34.559988+0000","last_active":"2024-04-28T12:48:41.637643+0000","last_peered":"2024-04-28T12:48:41.637643+0000","last_clean":"2024-04-28T12:48:41.637643+0000","last_became_active":"2024-04-28T12:48:34.559657+0000","last_became_peered":"2024-04-28T12:48:34.559657+0000","last_unstale":"2024-04-28T12:48:41.637643+0000","last_undegraded":"2024-04-28T12:48:41.637643+0000","last_fullsized":"2024-04-28T12:48:41.637643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:13:27.610212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638446+0000","last_change":"2024-04-28T12:48:34.553539+0000","last_active":"2024-04-28T12:48:41.638446+0000","last_peered":"2024-04-28T12:48:41.638446+0000","last_clean":"2024-04-28T12:48:41.638446+0000","last_became_active":"2024-04-28T12:48:34.553017+0000","last_became_peered":"2024-04-28T12:48:34.553017+0000","last_unstale":"2024-04-28T12:48:41.638446+0000","last_undegraded":"2024-04-28T12:48:41.638446+0000","last_fullsized":"2024-04-28T12:48:41.638446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:31:14.895894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638328+0000","last_change":"2024-04-28T12:48:34.553305+0000","last_active":"2024-04-28T12:48:41.638328+0000","last_peered":"2024-04-28T12:48:41.638328+0000","last_clean":"2024-04-28T12:48:41.638328+0000","last_became_active":"2024-04-28T12:48:34.552950+0000","last_became_peered":"2024-04-28T12:48:34.552950+0000","last_unstale":"2024-04-28T12:48:41.638328+0000","last_undegraded":"2024-04-28T12:48:41.638328+0000","last_fullsized":"2024-04-28T12:48:41.638328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:01:43.903986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638745+0000","last_change":"2024-04-28T12:48:34.558899+0000","last_active":"2024-04-28T12:48:41.638745+0000","last_peered":"2024-04-28T12:48:41.638745+0000","last_clean":"2024-04-28T12:48:41.638745+0000","last_became_active":"2024-04-28T12:48:34.558507+0000","last_became_peered":"2024-04-28T12:48:34.558507+0000","last_unstale":"2024-04-28T12:48:41.638745+0000","last_undegraded":"2024-04-28T12:48:41.638745+0000","last_fullsized":"2024-04-28T12:48:41.638745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:46:28.059231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576844+0000","last_change":"2024-04-28T12:48:34.545488+0000","last_active":"2024-04-28T12:48:40.576844+0000","last_peered":"2024-04-28T12:48:40.576844+0000","last_clean":"2024-04-28T12:48:40.576844+0000","last_became_active":"2024-04-28T12:48:34.545168+0000","last_became_peered":"2024-04-28T12:48:34.545168+0000","last_unstale":"2024-04-28T12:48:40.576844+0000","last_undegraded":"2024-04-28T12:48:40.576844+0000","last_fullsized":"2024-04-28T12:48:40.576844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:18:02.174760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639732+0000","last_change":"2024-04-28T12:48:34.544213+0000","last_active":"2024-04-28T12:48:41.639732+0000","last_peered":"2024-04-28T12:48:41.639732+0000","last_clean":"2024-04-28T12:48:41.639732+0000","last_became_active":"2024-04-28T12:48:34.543910+0000","last_became_peered":"2024-04-28T12:48:34.543910+0000","last_unstale":"2024-04-28T12:48:41.639732+0000","last_undegraded":"2024-04-28T12:48:41.639732+0000","last_fullsized":"2024-04-28T12:48:41.639732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:09:52.602695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639075+0000","last_change":"2024-04-28T12:48:34.548121+0000","last_active":"2024-04-28T12:48:41.639075+0000","last_peered":"2024-04-28T12:48:41.639075+0000","last_clean":"2024-04-28T12:48:41.639075+0000","last_became_active":"2024-04-28T12:48:34.547822+0000","last_became_peered":"2024-04-28T12:48:34.547822+0000","last_unstale":"2024-04-28T12:48:41.639075+0000","last_undegraded":"2024-04-28T12:48:41.639075+0000","last_fullsized":"2024-04-28T12:48:41.639075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T12:49:39.838959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639271+0000","last_change":"2024-04-28T12:48:34.544146+0000","last_active":"2024-04-28T12:48:41.639271+0000","last_peered":"2024-04-28T12:48:41.639271+0000","last_clean":"2024-04-28T12:48:41.639271+0000","last_became_active":"2024-04-28T12:48:34.543805+0000","last_became_peered":"2024-04-28T12:48:34.543805+0000","last_unstale":"2024-04-28T12:48:41.639271+0000","last_undegraded":"2024-04-28T12:48:41.639271+0000","last_fullsized":"2024-04-28T12:48:41.639271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:06:46.871316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579249+0000","last_change":"2024-04-28T12:48:34.536765+0000","last_active":"2024-04-28T12:48:40.579249+0000","last_peered":"2024-04-28T12:48:40.579249+0000","last_clean":"2024-04-28T12:48:40.579249+0000","last_became_active":"2024-04-28T12:48:34.535398+0000","last_became_peered":"2024-04-28T12:48:34.535398+0000","last_unstale":"2024-04-28T12:48:40.579249+0000","last_undegraded":"2024-04-28T12:48:40.579249+0000","last_fullsized":"2024-04-28T12:48:40.579249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:54:25.100414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639862+0000","last_change":"2024-04-28T12:48:34.573782+0000","last_active":"2024-04-28T12:48:41.639862+0000","last_peered":"2024-04-28T12:48:41.639862+0000","last_clean":"2024-04-28T12:48:41.639862+0000","last_became_active":"2024-04-28T12:48:34.573361+0000","last_became_peered":"2024-04-28T12:48:34.573361+0000","last_unstale":"2024-04-28T12:48:41.639862+0000","last_undegraded":"2024-04-28T12:48:41.639862+0000","last_fullsized":"2024-04-28T12:48:41.639862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:29:02.746038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639745+0000","last_change":"2024-04-28T12:48:34.570810+0000","last_active":"2024-04-28T12:48:41.639745+0000","last_peered":"2024-04-28T12:48:41.639745+0000","last_clean":"2024-04-28T12:48:41.639745+0000","last_became_active":"2024-04-28T12:48:34.570337+0000","last_became_peered":"2024-04-28T12:48:34.570337+0000","last_unstale":"2024-04-28T12:48:41.639745+0000","last_undegraded":"2024-04-28T12:48:41.639745+0000","last_fullsized":"2024-04-28T12:48:41.639745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:41:36.028710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576249+0000","last_change":"2024-04-28T12:48:34.558802+0000","last_active":"2024-04-28T12:48:40.576249+0000","last_peered":"2024-04-28T12:48:40.576249+0000","last_clean":"2024-04-28T12:48:40.576249+0000","last_became_active":"2024-04-28T12:48:34.558425+0000","last_became_peered":"2024-04-28T12:48:34.558425+0000","last_unstale":"2024-04-28T12:48:40.576249+0000","last_undegraded":"2024-04-28T12:48:40.576249+0000","last_fullsized":"2024-04-28T12:48:40.576249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:03:22.681965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639518+0000","last_change":"2024-04-28T12:48:34.569546+0000","last_active":"2024-04-28T12:48:41.639518+0000","last_peered":"2024-04-28T12:48:41.639518+0000","last_clean":"2024-04-28T12:48:41.639518+0000","last_became_active":"2024-04-28T12:48:34.569231+0000","last_became_peered":"2024-04-28T12:48:34.569231+0000","last_unstale":"2024-04-28T12:48:41.639518+0000","last_undegraded":"2024-04-28T12:48:41.639518+0000","last_fullsized":"2024-04-28T12:48:41.639518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:19:24.491204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.524232+0000","last_change":"2024-04-28T12:48:34.555634+0000","last_active":"2024-04-28T12:48:41.524232+0000","last_peered":"2024-04-28T12:48:41.524232+0000","last_clean":"2024-04-28T12:48:41.524232+0000","last_became_active":"2024-04-28T12:48:34.555054+0000","last_became_peered":"2024-04-28T12:48:34.555054+0000","last_unstale":"2024-04-28T12:48:41.524232+0000","last_undegraded":"2024-04-28T12:48:41.524232+0000","last_fullsized":"2024-04-28T12:48:41.524232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:37.493994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639094+0000","last_change":"2024-04-28T12:48:34.568857+0000","last_active":"2024-04-28T12:48:41.639094+0000","last_peered":"2024-04-28T12:48:41.639094+0000","last_clean":"2024-04-28T12:48:41.639094+0000","last_became_active":"2024-04-28T12:48:34.568550+0000","last_became_peered":"2024-04-28T12:48:34.568550+0000","last_unstale":"2024-04-28T12:48:41.639094+0000","last_undegraded":"2024-04-28T12:48:41.639094+0000","last_fullsized":"2024-04-28T12:48:41.639094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:09:40.709446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638885+0000","last_change":"2024-04-28T12:48:39.608021+0000","last_active":"2024-04-28T12:48:41.638885+0000","last_peered":"2024-04-28T12:48:41.638885+0000","last_clean":"2024-04-28T12:48:41.638885+0000","last_became_active":"2024-04-28T12:48:39.607187+0000","last_became_peered":"2024-04-28T12:48:39.607187+0000","last_unstale":"2024-04-28T12:48:41.638885+0000","last_undegraded":"2024-04-28T12:48:41.638885+0000","last_fullsized":"2024-04-28T12:48:41.638885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:31:26.346414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.524409+0000","last_change":"2024-04-28T12:48:33.839179+0000","last_active":"2024-04-28T12:48:41.524409+0000","last_peered":"2024-04-28T12:48:41.524409+0000","last_clean":"2024-04-28T12:48:41.524409+0000","last_became_active":"2024-04-28T12:48:33.838004+0000","last_became_peered":"2024-04-28T12:48:33.838004+0000","last_unstale":"2024-04-28T12:48:41.524409+0000","last_undegraded":"2024-04-28T12:48:41.524409+0000","last_fullsized":"2024-04-28T12:48:41.524409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:32:14.765652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.641375+0000","last_change":"2024-04-28T12:48:34.571840+0000","last_active":"2024-04-28T12:48:41.641375+0000","last_peered":"2024-04-28T12:48:41.641375+0000","last_clean":"2024-04-28T12:48:41.641375+0000","last_became_active":"2024-04-28T12:48:34.568001+0000","last_became_peered":"2024-04-28T12:48:34.568001+0000","last_unstale":"2024-04-28T12:48:41.641375+0000","last_undegraded":"2024-04-28T12:48:41.641375+0000","last_fullsized":"2024-04-28T12:48:41.641375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:15:22.276123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638570+0000","last_change":"2024-04-28T12:48:39.605055+0000","last_active":"2024-04-28T12:48:41.638570+0000","last_peered":"2024-04-28T12:48:41.638570+0000","last_clean":"2024-04-28T12:48:41.638570+0000","last_became_active":"2024-04-28T12:48:39.604566+0000","last_became_peered":"2024-04-28T12:48:39.604566+0000","last_unstale":"2024-04-28T12:48:41.638570+0000","last_undegraded":"2024-04-28T12:48:41.638570+0000","last_fullsized":"2024-04-28T12:48:41.638570+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:48.945467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.641876+0000","last_change":"2024-04-28T12:48:33.841745+0000","last_active":"2024-04-28T12:48:41.641876+0000","last_peered":"2024-04-28T12:48:41.641876+0000","last_clean":"2024-04-28T12:48:41.641876+0000","last_became_active":"2024-04-28T12:48:33.835003+0000","last_became_peered":"2024-04-28T12:48:33.835003+0000","last_unstale":"2024-04-28T12:48:41.641876+0000","last_undegraded":"2024-04-28T12:48:41.641876+0000","last_fullsized":"2024-04-28T12:48:41.641876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:40:14.253322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.641199+0000","last_change":"2024-04-28T12:48:34.577688+0000","last_active":"2024-04-28T12:48:41.641199+0000","last_peered":"2024-04-28T12:48:41.641199+0000","last_clean":"2024-04-28T12:48:41.641199+0000","last_became_active":"2024-04-28T12:48:34.577269+0000","last_became_peered":"2024-04-28T12:48:34.577269+0000","last_unstale":"2024-04-28T12:48:41.641199+0000","last_undegraded":"2024-04-28T12:48:41.641199+0000","last_fullsized":"2024-04-28T12:48:41.641199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:30:13.744015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638617+0000","last_change":"2024-04-28T12:48:39.600708+0000","last_active":"2024-04-28T12:48:41.638617+0000","last_peered":"2024-04-28T12:48:41.638617+0000","last_clean":"2024-04-28T12:48:41.638617+0000","last_became_active":"2024-04-28T12:48:39.600257+0000","last_became_peered":"2024-04-28T12:48:39.600257+0000","last_unstale":"2024-04-28T12:48:41.638617+0000","last_undegraded":"2024-04-28T12:48:41.638617+0000","last_fullsized":"2024-04-28T12:48:41.638617+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:08:02.685278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638387+0000","last_change":"2024-04-28T12:48:33.837824+0000","last_active":"2024-04-28T12:48:41.638387+0000","last_peered":"2024-04-28T12:48:41.638387+0000","last_clean":"2024-04-28T12:48:41.638387+0000","last_became_active":"2024-04-28T12:48:33.837505+0000","last_became_peered":"2024-04-28T12:48:33.837505+0000","last_unstale":"2024-04-28T12:48:41.638387+0000","last_undegraded":"2024-04-28T12:48:41.638387+0000","last_fullsized":"2024-04-28T12:48:41.638387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:21:10.677381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576533+0000","last_change":"2024-04-28T12:48:34.554575+0000","last_active":"2024-04-28T12:48:40.576533+0000","last_peered":"2024-04-28T12:48:40.576533+0000","last_clean":"2024-04-28T12:48:40.576533+0000","last_became_active":"2024-04-28T12:48:34.554214+0000","last_became_peered":"2024-04-28T12:48:34.554214+0000","last_unstale":"2024-04-28T12:48:40.576533+0000","last_undegraded":"2024-04-28T12:48:40.576533+0000","last_fullsized":"2024-04-28T12:48:40.576533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:37:03.369874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638017+0000","last_change":"2024-04-28T12:48:39.604793+0000","last_active":"2024-04-28T12:48:41.638017+0000","last_peered":"2024-04-28T12:48:41.638017+0000","last_clean":"2024-04-28T12:48:41.638017+0000","last_became_active":"2024-04-28T12:48:39.604338+0000","last_became_peered":"2024-04-28T12:48:39.604338+0000","last_unstale":"2024-04-28T12:48:41.638017+0000","last_undegraded":"2024-04-28T12:48:41.638017+0000","last_fullsized":"2024-04-28T12:48:41.638017+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:14:56.225098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637754+0000","last_change":"2024-04-28T12:48:33.839375+0000","last_active":"2024-04-28T12:48:41.637754+0000","last_peered":"2024-04-28T12:48:41.637754+0000","last_clean":"2024-04-28T12:48:41.637754+0000","last_became_active":"2024-04-28T12:48:33.839062+0000","last_became_peered":"2024-04-28T12:48:33.839062+0000","last_unstale":"2024-04-28T12:48:41.637754+0000","last_undegraded":"2024-04-28T12:48:41.637754+0000","last_fullsized":"2024-04-28T12:48:41.637754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:20:06.692743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576546+0000","last_change":"2024-04-28T12:48:34.543341+0000","last_active":"2024-04-28T12:48:40.576546+0000","last_peered":"2024-04-28T12:48:40.576546+0000","last_clean":"2024-04-28T12:48:40.576546+0000","last_became_active":"2024-04-28T12:48:34.543033+0000","last_became_peered":"2024-04-28T12:48:34.543033+0000","last_unstale":"2024-04-28T12:48:40.576546+0000","last_undegraded":"2024-04-28T12:48:40.576546+0000","last_fullsized":"2024-04-28T12:48:40.576546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:41:27.654577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638950+0000","last_change":"2024-04-28T12:48:39.589856+0000","last_active":"2024-04-28T12:48:41.638950+0000","last_peered":"2024-04-28T12:48:41.638950+0000","last_clean":"2024-04-28T12:48:41.638950+0000","last_became_active":"2024-04-28T12:48:39.589148+0000","last_became_peered":"2024-04-28T12:48:39.589148+0000","last_unstale":"2024-04-28T12:48:41.638950+0000","last_undegraded":"2024-04-28T12:48:41.638950+0000","last_fullsized":"2024-04-28T12:48:41.638950+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:34:11.897109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:08.751314+0000","last_change":"2024-04-28T12:48:33.829387+0000","last_active":"2024-04-28T12:49:08.751314+0000","last_peered":"2024-04-28T12:49:08.751314+0000","last_clean":"2024-04-28T12:49:08.751314+0000","last_became_active":"2024-04-28T12:48:33.829011+0000","last_became_peered":"2024-04-28T12:48:33.829011+0000","last_unstale":"2024-04-28T12:49:08.751314+0000","last_undegraded":"2024-04-28T12:49:08.751314+0000","last_fullsized":"2024-04-28T12:49:08.751314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:54:22.096246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637534+0000","last_change":"2024-04-28T12:48:34.554629+0000","last_active":"2024-04-28T12:48:41.637534+0000","last_peered":"2024-04-28T12:48:41.637534+0000","last_clean":"2024-04-28T12:48:41.637534+0000","last_became_active":"2024-04-28T12:48:34.554231+0000","last_became_peered":"2024-04-28T12:48:34.554231+0000","last_unstale":"2024-04-28T12:48:41.637534+0000","last_undegraded":"2024-04-28T12:48:41.637534+0000","last_fullsized":"2024-04-28T12:48:41.637534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:21:53.706120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638245+0000","last_change":"2024-04-28T12:48:39.606416+0000","last_active":"2024-04-28T12:48:41.638245+0000","last_peered":"2024-04-28T12:48:41.638245+0000","last_clean":"2024-04-28T12:48:41.638245+0000","last_became_active":"2024-04-28T12:48:39.605957+0000","last_became_peered":"2024-04-28T12:48:39.605957+0000","last_unstale":"2024-04-28T12:48:41.638245+0000","last_undegraded":"2024-04-28T12:48:41.638245+0000","last_fullsized":"2024-04-28T12:48:41.638245+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:39:16.315446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:07.739900+0000","last_change":"2024-04-28T12:48:33.841119+0000","last_active":"2024-04-28T12:49:07.739900+0000","last_peered":"2024-04-28T12:49:07.739900+0000","last_clean":"2024-04-28T12:49:07.739900+0000","last_became_active":"2024-04-28T12:48:33.840833+0000","last_became_peered":"2024-04-28T12:48:33.840833+0000","last_unstale":"2024-04-28T12:49:07.739900+0000","last_undegraded":"2024-04-28T12:49:07.739900+0000","last_fullsized":"2024-04-28T12:49:07.739900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:49:31.145968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576510+0000","last_change":"2024-04-28T12:48:34.565140+0000","last_active":"2024-04-28T12:48:40.576510+0000","last_peered":"2024-04-28T12:48:40.576510+0000","last_clean":"2024-04-28T12:48:40.576510+0000","last_became_active":"2024-04-28T12:48:34.564692+0000","last_became_peered":"2024-04-28T12:48:34.564692+0000","last_unstale":"2024-04-28T12:48:40.576510+0000","last_undegraded":"2024-04-28T12:48:40.576510+0000","last_fullsized":"2024-04-28T12:48:40.576510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:55:27.440482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639430+0000","last_change":"2024-04-28T12:48:39.611109+0000","last_active":"2024-04-28T12:48:41.639430+0000","last_peered":"2024-04-28T12:48:41.639430+0000","last_clean":"2024-04-28T12:48:41.639430+0000","last_became_active":"2024-04-28T12:48:39.610754+0000","last_became_peered":"2024-04-28T12:48:39.610754+0000","last_unstale":"2024-04-28T12:48:41.639430+0000","last_undegraded":"2024-04-28T12:48:41.639430+0000","last_fullsized":"2024-04-28T12:48:41.639430+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:55:14.610718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:07.742515+0000","last_change":"2024-04-28T12:48:33.838037+0000","last_active":"2024-04-28T12:49:07.742515+0000","last_peered":"2024-04-28T12:49:07.742515+0000","last_clean":"2024-04-28T12:49:07.742515+0000","last_became_active":"2024-04-28T12:48:33.837704+0000","last_became_peered":"2024-04-28T12:48:33.837704+0000","last_unstale":"2024-04-28T12:49:07.742515+0000","last_undegraded":"2024-04-28T12:49:07.742515+0000","last_fullsized":"2024-04-28T12:49:07.742515+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:46:24.595855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638948+0000","last_change":"2024-04-28T12:48:34.558165+0000","last_active":"2024-04-28T12:48:41.638948+0000","last_peered":"2024-04-28T12:48:41.638948+0000","last_clean":"2024-04-28T12:48:41.638948+0000","last_became_active":"2024-04-28T12:48:34.557728+0000","last_became_peered":"2024-04-28T12:48:34.557728+0000","last_unstale":"2024-04-28T12:48:41.638948+0000","last_undegraded":"2024-04-28T12:48:41.638948+0000","last_fullsized":"2024-04-28T12:48:41.638948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:09:00.169634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640679+0000","last_change":"2024-04-28T12:48:39.615515+0000","last_active":"2024-04-28T12:48:41.640679+0000","last_peered":"2024-04-28T12:48:41.640679+0000","last_clean":"2024-04-28T12:48:41.640679+0000","last_became_active":"2024-04-28T12:48:39.614717+0000","last_became_peered":"2024-04-28T12:48:39.614717+0000","last_unstale":"2024-04-28T12:48:41.640679+0000","last_undegraded":"2024-04-28T12:48:41.640679+0000","last_fullsized":"2024-04-28T12:48:41.640679+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:11:06.633399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.575969+0000","last_change":"2024-04-28T12:48:33.832694+0000","last_active":"2024-04-28T12:48:40.575969+0000","last_peered":"2024-04-28T12:48:40.575969+0000","last_clean":"2024-04-28T12:48:40.575969+0000","last_became_active":"2024-04-28T12:48:33.832381+0000","last_became_peered":"2024-04-28T12:48:33.832381+0000","last_unstale":"2024-04-28T12:48:40.575969+0000","last_undegraded":"2024-04-28T12:48:40.575969+0000","last_fullsized":"2024-04-28T12:48:40.575969+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:56:57.511053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.523396+0000","last_change":"2024-04-28T12:48:34.560229+0000","last_active":"2024-04-28T12:48:41.523396+0000","last_peered":"2024-04-28T12:48:41.523396+0000","last_clean":"2024-04-28T12:48:41.523396+0000","last_became_active":"2024-04-28T12:48:34.559926+0000","last_became_peered":"2024-04-28T12:48:34.559926+0000","last_unstale":"2024-04-28T12:48:41.523396+0000","last_undegraded":"2024-04-28T12:48:41.523396+0000","last_fullsized":"2024-04-28T12:48:41.523396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:38:22.476449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638729+0000","last_change":"2024-04-28T12:48:39.604042+0000","last_active":"2024-04-28T12:48:41.638729+0000","last_peered":"2024-04-28T12:48:41.638729+0000","last_clean":"2024-04-28T12:48:41.638729+0000","last_became_active":"2024-04-28T12:48:39.603631+0000","last_became_peered":"2024-04-28T12:48:39.603631+0000","last_unstale":"2024-04-28T12:48:41.638729+0000","last_undegraded":"2024-04-28T12:48:41.638729+0000","last_fullsized":"2024-04-28T12:48:41.638729+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:53:20.073434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.728038+0000","last_change":"2024-04-28T12:48:33.846372+0000","last_active":"2024-04-28T12:49:06.728038+0000","last_peered":"2024-04-28T12:49:06.728038+0000","last_clean":"2024-04-28T12:49:06.728038+0000","last_became_active":"2024-04-28T12:48:33.846138+0000","last_became_peered":"2024-04-28T12:48:33.846138+0000","last_unstale":"2024-04-28T12:49:06.728038+0000","last_undegraded":"2024-04-28T12:49:06.728038+0000","last_fullsized":"2024-04-28T12:49:06.728038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:46:45.889188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576817+0000","last_change":"2024-04-28T12:48:34.554872+0000","last_active":"2024-04-28T12:48:40.576817+0000","last_peered":"2024-04-28T12:48:40.576817+0000","last_clean":"2024-04-28T12:48:40.576817+0000","last_became_active":"2024-04-28T12:48:34.554555+0000","last_became_peered":"2024-04-28T12:48:34.554555+0000","last_unstale":"2024-04-28T12:48:40.576817+0000","last_undegraded":"2024-04-28T12:48:40.576817+0000","last_fullsized":"2024-04-28T12:48:40.576817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:17:51.280686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.523429+0000","last_change":"2024-04-28T12:48:39.603114+0000","last_active":"2024-04-28T12:48:41.523429+0000","last_peered":"2024-04-28T12:48:41.523429+0000","last_clean":"2024-04-28T12:48:41.523429+0000","last_became_active":"2024-04-28T12:48:39.598404+0000","last_became_peered":"2024-04-28T12:48:39.598404+0000","last_unstale":"2024-04-28T12:48:41.523429+0000","last_undegraded":"2024-04-28T12:48:41.523429+0000","last_fullsized":"2024-04-28T12:48:41.523429+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:23.824317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.727506+0000","last_change":"2024-04-28T12:48:33.831626+0000","last_active":"2024-04-28T12:49:06.727506+0000","last_peered":"2024-04-28T12:49:06.727506+0000","last_clean":"2024-04-28T12:49:06.727506+0000","last_became_active":"2024-04-28T12:48:33.831294+0000","last_became_peered":"2024-04-28T12:48:33.831294+0000","last_unstale":"2024-04-28T12:49:06.727506+0000","last_undegraded":"2024-04-28T12:49:06.727506+0000","last_fullsized":"2024-04-28T12:49:06.727506+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:58:43.775695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640929+0000","last_change":"2024-04-28T12:48:34.574987+0000","last_active":"2024-04-28T12:48:41.640929+0000","last_peered":"2024-04-28T12:48:41.640929+0000","last_clean":"2024-04-28T12:48:41.640929+0000","last_became_active":"2024-04-28T12:48:34.574668+0000","last_became_peered":"2024-04-28T12:48:34.574668+0000","last_unstale":"2024-04-28T12:48:41.640929+0000","last_undegraded":"2024-04-28T12:48:41.640929+0000","last_fullsized":"2024-04-28T12:48:41.640929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:24:06.195671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638944+0000","last_change":"2024-04-28T12:48:39.600246+0000","last_active":"2024-04-28T12:48:41.638944+0000","last_peered":"2024-04-28T12:48:41.638944+0000","last_clean":"2024-04-28T12:48:41.638944+0000","last_became_active":"2024-04-28T12:48:39.599811+0000","last_became_peered":"2024-04-28T12:48:39.599811+0000","last_unstale":"2024-04-28T12:48:41.638944+0000","last_undegraded":"2024-04-28T12:48:41.638944+0000","last_fullsized":"2024-04-28T12:48:41.638944+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:32:54.888851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.728380+0000","last_change":"2024-04-28T12:48:33.840884+0000","last_active":"2024-04-28T12:49:06.728380+0000","last_peered":"2024-04-28T12:49:06.728380+0000","last_clean":"2024-04-28T12:49:06.728380+0000","last_became_active":"2024-04-28T12:48:33.840467+0000","last_became_peered":"2024-04-28T12:48:33.840467+0000","last_unstale":"2024-04-28T12:49:06.728380+0000","last_undegraded":"2024-04-28T12:49:06.728380+0000","last_fullsized":"2024-04-28T12:49:06.728380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:28:16.480520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576667+0000","last_change":"2024-04-28T12:48:34.545000+0000","last_active":"2024-04-28T12:48:40.576667+0000","last_peered":"2024-04-28T12:48:40.576667+0000","last_clean":"2024-04-28T12:48:40.576667+0000","last_became_active":"2024-04-28T12:48:34.544662+0000","last_became_peered":"2024-04-28T12:48:34.544662+0000","last_unstale":"2024-04-28T12:48:40.576667+0000","last_undegraded":"2024-04-28T12:48:40.576667+0000","last_fullsized":"2024-04-28T12:48:40.576667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:19:21.640362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.523534+0000","last_change":"2024-04-28T12:48:39.596755+0000","last_active":"2024-04-28T12:48:41.523534+0000","last_peered":"2024-04-28T12:48:41.523534+0000","last_clean":"2024-04-28T12:48:41.523534+0000","last_became_active":"2024-04-28T12:48:39.596292+0000","last_became_peered":"2024-04-28T12:48:39.596292+0000","last_unstale":"2024-04-28T12:48:41.523534+0000","last_undegraded":"2024-04-28T12:48:41.523534+0000","last_fullsized":"2024-04-28T12:48:41.523534+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:21:27.026940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'16","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:09.241584+0000","last_change":"2024-04-28T12:48:33.833840+0000","last_active":"2024-04-28T12:49:09.241584+0000","last_peered":"2024-04-28T12:49:09.241584+0000","last_clean":"2024-04-28T12:49:09.241584+0000","last_became_active":"2024-04-28T12:48:33.833536+0000","last_became_peered":"2024-04-28T12:48:33.833536+0000","last_unstale":"2024-04-28T12:49:09.241584+0000","last_undegraded":"2024-04-28T12:49:09.241584+0000","last_fullsized":"2024-04-28T12:49:09.241584+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:37:01.455373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10442,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638084+0000","last_change":"2024-04-28T12:48:34.548046+0000","last_active":"2024-04-28T12:48:41.638084+0000","last_peered":"2024-04-28T12:48:41.638084+0000","last_clean":"2024-04-28T12:48:41.638084+0000","last_became_active":"2024-04-28T12:48:34.547724+0000","last_became_peered":"2024-04-28T12:48:34.547724+0000","last_unstale":"2024-04-28T12:48:41.638084+0000","last_undegraded":"2024-04-28T12:48:41.638084+0000","last_fullsized":"2024-04-28T12:48:41.638084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:07.287597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576941+0000","last_change":"2024-04-28T12:48:39.602976+0000","last_active":"2024-04-28T12:48:40.576941+0000","last_peered":"2024-04-28T12:48:40.576941+0000","last_clean":"2024-04-28T12:48:40.576941+0000","last_became_active":"2024-04-28T12:48:39.602588+0000","last_became_peered":"2024-04-28T12:48:39.602588+0000","last_unstale":"2024-04-28T12:48:40.576941+0000","last_undegraded":"2024-04-28T12:48:40.576941+0000","last_fullsized":"2024-04-28T12:48:40.576941+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:18:19.549434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:07.741715+0000","last_change":"2024-04-28T12:48:33.831276+0000","last_active":"2024-04-28T12:49:07.741715+0000","last_peered":"2024-04-28T12:49:07.741715+0000","last_clean":"2024-04-28T12:49:07.741715+0000","last_became_active":"2024-04-28T12:48:33.830845+0000","last_became_peered":"2024-04-28T12:48:33.830845+0000","last_unstale":"2024-04-28T12:49:07.741715+0000","last_undegraded":"2024-04-28T12:49:07.741715+0000","last_fullsized":"2024-04-28T12:49:07.741715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:21:02.619309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.578481+0000","last_change":"2024-04-28T12:48:34.566767+0000","last_active":"2024-04-28T12:48:40.578481+0000","last_peered":"2024-04-28T12:48:40.578481+0000","last_clean":"2024-04-28T12:48:40.578481+0000","last_became_active":"2024-04-28T12:48:34.566404+0000","last_became_peered":"2024-04-28T12:48:34.566404+0000","last_unstale":"2024-04-28T12:48:40.578481+0000","last_undegraded":"2024-04-28T12:48:40.578481+0000","last_fullsized":"2024-04-28T12:48:40.578481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:49:32.684046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639425+0000","last_change":"2024-04-28T12:48:39.607471+0000","last_active":"2024-04-28T12:48:41.639425+0000","last_peered":"2024-04-28T12:48:41.639425+0000","last_clean":"2024-04-28T12:48:41.639425+0000","last_became_active":"2024-04-28T12:48:39.607040+0000","last_became_peered":"2024-04-28T12:48:39.607040+0000","last_unstale":"2024-04-28T12:48:41.639425+0000","last_undegraded":"2024-04-28T12:48:41.639425+0000","last_fullsized":"2024-04-28T12:48:41.639425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:06:32.613106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.729004+0000","last_change":"2024-04-28T12:48:33.832519+0000","last_active":"2024-04-28T12:49:06.729004+0000","last_peered":"2024-04-28T12:49:06.729004+0000","last_clean":"2024-04-28T12:49:06.729004+0000","last_became_active":"2024-04-28T12:48:33.832146+0000","last_became_peered":"2024-04-28T12:48:33.832146+0000","last_unstale":"2024-04-28T12:49:06.729004+0000","last_undegraded":"2024-04-28T12:49:06.729004+0000","last_fullsized":"2024-04-28T12:49:06.729004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:02:48.755990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638522+0000","last_change":"2024-04-28T12:48:34.557093+0000","last_active":"2024-04-28T12:48:41.638522+0000","last_peered":"2024-04-28T12:48:41.638522+0000","last_clean":"2024-04-28T12:48:41.638522+0000","last_became_active":"2024-04-28T12:48:34.556767+0000","last_became_peered":"2024-04-28T12:48:34.556767+0000","last_unstale":"2024-04-28T12:48:41.638522+0000","last_undegraded":"2024-04-28T12:48:41.638522+0000","last_fullsized":"2024-04-28T12:48:41.638522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:45:48.634998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.524332+0000","last_change":"2024-04-28T12:48:39.612227+0000","last_active":"2024-04-28T12:48:41.524332+0000","last_peered":"2024-04-28T12:48:41.524332+0000","last_clean":"2024-04-28T12:48:41.524332+0000","last_became_active":"2024-04-28T12:48:39.611065+0000","last_became_peered":"2024-04-28T12:48:39.611065+0000","last_unstale":"2024-04-28T12:48:41.524332+0000","last_undegraded":"2024-04-28T12:48:41.524332+0000","last_fullsized":"2024-04-28T12:48:41.524332+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:15:55.833369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:09.752917+0000","last_change":"2024-04-28T12:48:33.835102+0000","last_active":"2024-04-28T12:49:09.752917+0000","last_peered":"2024-04-28T12:49:09.752917+0000","last_clean":"2024-04-28T12:49:09.752917+0000","last_became_active":"2024-04-28T12:48:33.834672+0000","last_became_peered":"2024-04-28T12:48:33.834672+0000","last_unstale":"2024-04-28T12:49:09.752917+0000","last_undegraded":"2024-04-28T12:49:09.752917+0000","last_fullsized":"2024-04-28T12:49:09.752917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:41:39.388880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637994+0000","last_change":"2024-04-28T12:48:34.548845+0000","last_active":"2024-04-28T12:48:41.637994+0000","last_peered":"2024-04-28T12:48:41.637994+0000","last_clean":"2024-04-28T12:48:41.637994+0000","last_became_active":"2024-04-28T12:48:34.548546+0000","last_became_peered":"2024-04-28T12:48:34.548546+0000","last_unstale":"2024-04-28T12:48:41.637994+0000","last_undegraded":"2024-04-28T12:48:41.637994+0000","last_fullsized":"2024-04-28T12:48:41.637994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:00:53.848671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579467+0000","last_change":"2024-04-28T12:48:39.613039+0000","last_active":"2024-04-28T12:48:40.579467+0000","last_peered":"2024-04-28T12:48:40.579467+0000","last_clean":"2024-04-28T12:48:40.579467+0000","last_became_active":"2024-04-28T12:48:39.609383+0000","last_became_peered":"2024-04-28T12:48:39.609383+0000","last_unstale":"2024-04-28T12:48:40.579467+0000","last_undegraded":"2024-04-28T12:48:40.579467+0000","last_fullsized":"2024-04-28T12:48:40.579467+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:55:47.031973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'10","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.728543+0000","last_change":"2024-04-28T12:48:33.843087+0000","last_active":"2024-04-28T12:49:06.728543+0000","last_peered":"2024-04-28T12:49:06.728543+0000","last_clean":"2024-04-28T12:49:06.728543+0000","last_became_active":"2024-04-28T12:48:33.842775+0000","last_became_peered":"2024-04-28T12:48:33.842775+0000","last_unstale":"2024-04-28T12:49:06.728543+0000","last_undegraded":"2024-04-28T12:49:06.728543+0000","last_fullsized":"2024-04-28T12:49:06.728543+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:39:00.797781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576095+0000","last_change":"2024-04-28T12:48:34.544373+0000","last_active":"2024-04-28T12:48:40.576095+0000","last_peered":"2024-04-28T12:48:40.576095+0000","last_clean":"2024-04-28T12:48:40.576095+0000","last_became_active":"2024-04-28T12:48:34.544040+0000","last_became_peered":"2024-04-28T12:48:34.544040+0000","last_unstale":"2024-04-28T12:48:40.576095+0000","last_undegraded":"2024-04-28T12:48:40.576095+0000","last_fullsized":"2024-04-28T12:48:40.576095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:52:51.088454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576114+0000","last_change":"2024-04-28T12:48:39.608191+0000","last_active":"2024-04-28T12:48:40.576114+0000","last_peered":"2024-04-28T12:48:40.576114+0000","last_clean":"2024-04-28T12:48:40.576114+0000","last_became_active":"2024-04-28T12:48:39.607724+0000","last_became_peered":"2024-04-28T12:48:39.607724+0000","last_unstale":"2024-04-28T12:48:40.576114+0000","last_undegraded":"2024-04-28T12:48:40.576114+0000","last_fullsized":"2024-04-28T12:48:40.576114+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:19:58.040666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638661+0000","last_change":"2024-04-28T12:48:33.839588+0000","last_active":"2024-04-28T12:48:41.638661+0000","last_peered":"2024-04-28T12:48:41.638661+0000","last_clean":"2024-04-28T12:48:41.638661+0000","last_became_active":"2024-04-28T12:48:33.839094+0000","last_became_peered":"2024-04-28T12:48:33.839094+0000","last_unstale":"2024-04-28T12:48:41.638661+0000","last_undegraded":"2024-04-28T12:48:41.638661+0000","last_fullsized":"2024-04-28T12:48:41.638661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:18:35.824206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639495+0000","last_change":"2024-04-28T12:48:34.547354+0000","last_active":"2024-04-28T12:48:41.639495+0000","last_peered":"2024-04-28T12:48:41.639495+0000","last_clean":"2024-04-28T12:48:41.639495+0000","last_became_active":"2024-04-28T12:48:34.546688+0000","last_became_peered":"2024-04-28T12:48:34.546688+0000","last_unstale":"2024-04-28T12:48:41.639495+0000","last_undegraded":"2024-04-28T12:48:41.639495+0000","last_fullsized":"2024-04-28T12:48:41.639495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:10:23.333028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637748+0000","last_change":"2024-04-28T12:48:39.912611+0000","last_active":"2024-04-28T12:48:41.637748+0000","last_peered":"2024-04-28T12:48:41.637748+0000","last_clean":"2024-04-28T12:48:41.637748+0000","last_became_active":"2024-04-28T12:48:39.912097+0000","last_became_peered":"2024-04-28T12:48:39.912097+0000","last_unstale":"2024-04-28T12:48:41.637748+0000","last_undegraded":"2024-04-28T12:48:41.637748+0000","last_fullsized":"2024-04-28T12:48:41.637748+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:31:38.597920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.523912+0000","last_change":"2024-04-28T12:48:33.836343+0000","last_active":"2024-04-28T12:48:41.523912+0000","last_peered":"2024-04-28T12:48:41.523912+0000","last_clean":"2024-04-28T12:48:41.523912+0000","last_became_active":"2024-04-28T12:48:33.835216+0000","last_became_peered":"2024-04-28T12:48:33.835216+0000","last_unstale":"2024-04-28T12:48:41.523912+0000","last_undegraded":"2024-04-28T12:48:41.523912+0000","last_fullsized":"2024-04-28T12:48:41.523912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:44:39.248936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639279+0000","last_change":"2024-04-28T12:48:34.568892+0000","last_active":"2024-04-28T12:48:41.639279+0000","last_peered":"2024-04-28T12:48:41.639279+0000","last_clean":"2024-04-28T12:48:41.639279+0000","last_became_active":"2024-04-28T12:48:34.568581+0000","last_became_peered":"2024-04-28T12:48:34.568581+0000","last_unstale":"2024-04-28T12:48:41.639279+0000","last_undegraded":"2024-04-28T12:48:41.639279+0000","last_fullsized":"2024-04-28T12:48:41.639279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:56:51.661049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579187+0000","last_change":"2024-04-28T12:48:39.616095+0000","last_active":"2024-04-28T12:48:40.579187+0000","last_peered":"2024-04-28T12:48:40.579187+0000","last_clean":"2024-04-28T12:48:40.579187+0000","last_became_active":"2024-04-28T12:48:39.614891+0000","last_became_peered":"2024-04-28T12:48:39.614891+0000","last_unstale":"2024-04-28T12:48:40.579187+0000","last_undegraded":"2024-04-28T12:48:40.579187+0000","last_fullsized":"2024-04-28T12:48:40.579187+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:01:05.645329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638981+0000","last_change":"2024-04-28T12:48:33.833599+0000","last_active":"2024-04-28T12:48:41.638981+0000","last_peered":"2024-04-28T12:48:41.638981+0000","last_clean":"2024-04-28T12:48:41.638981+0000","last_became_active":"2024-04-28T12:48:33.833062+0000","last_became_peered":"2024-04-28T12:48:33.833062+0000","last_unstale":"2024-04-28T12:48:41.638981+0000","last_undegraded":"2024-04-28T12:48:41.638981+0000","last_fullsized":"2024-04-28T12:48:41.638981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:00:13.094007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638529+0000","last_change":"2024-04-28T12:48:34.558759+0000","last_active":"2024-04-28T12:48:41.638529+0000","last_peered":"2024-04-28T12:48:41.638529+0000","last_clean":"2024-04-28T12:48:41.638529+0000","last_became_active":"2024-04-28T12:48:34.558413+0000","last_became_peered":"2024-04-28T12:48:34.558413+0000","last_unstale":"2024-04-28T12:48:41.638529+0000","last_undegraded":"2024-04-28T12:48:41.638529+0000","last_fullsized":"2024-04-28T12:48:41.638529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:07:47.106653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579282+0000","last_change":"2024-04-28T12:48:39.914492+0000","last_active":"2024-04-28T12:48:40.579282+0000","last_peered":"2024-04-28T12:48:40.579282+0000","last_clean":"2024-04-28T12:48:40.579282+0000","last_became_active":"2024-04-28T12:48:39.913113+0000","last_became_peered":"2024-04-28T12:48:39.913113+0000","last_unstale":"2024-04-28T12:48:40.579282+0000","last_undegraded":"2024-04-28T12:48:40.579282+0000","last_fullsized":"2024-04-28T12:48:40.579282+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:49:21.221058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576450+0000","last_change":"2024-04-28T12:48:34.544121+0000","last_active":"2024-04-28T12:48:40.576450+0000","last_peered":"2024-04-28T12:48:40.576450+0000","last_clean":"2024-04-28T12:48:40.576450+0000","last_became_active":"2024-04-28T12:48:34.543816+0000","last_became_peered":"2024-04-28T12:48:34.543816+0000","last_unstale":"2024-04-28T12:48:40.576450+0000","last_undegraded":"2024-04-28T12:48:40.576450+0000","last_fullsized":"2024-04-28T12:48:40.576450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:24.237518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:08.749446+0000","last_change":"2024-04-28T12:48:33.845996+0000","last_active":"2024-04-28T12:49:08.749446+0000","last_peered":"2024-04-28T12:49:08.749446+0000","last_clean":"2024-04-28T12:49:08.749446+0000","last_became_active":"2024-04-28T12:48:33.845618+0000","last_became_peered":"2024-04-28T12:48:33.845618+0000","last_unstale":"2024-04-28T12:49:08.749446+0000","last_undegraded":"2024-04-28T12:49:08.749446+0000","last_fullsized":"2024-04-28T12:49:08.749446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:53:12.312461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638659+0000","last_change":"2024-04-28T12:48:39.589281+0000","last_active":"2024-04-28T12:48:41.638659+0000","last_peered":"2024-04-28T12:48:41.638659+0000","last_clean":"2024-04-28T12:48:41.638659+0000","last_became_active":"2024-04-28T12:48:39.588964+0000","last_became_peered":"2024-04-28T12:48:39.588964+0000","last_unstale":"2024-04-28T12:48:41.638659+0000","last_undegraded":"2024-04-28T12:48:41.638659+0000","last_fullsized":"2024-04-28T12:48:41.638659+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:32:12.039897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639751+0000","last_change":"2024-04-28T12:48:34.567062+0000","last_active":"2024-04-28T12:48:41.639751+0000","last_peered":"2024-04-28T12:48:41.639751+0000","last_clean":"2024-04-28T12:48:41.639751+0000","last_became_active":"2024-04-28T12:48:34.565977+0000","last_became_peered":"2024-04-28T12:48:34.565977+0000","last_unstale":"2024-04-28T12:48:41.639751+0000","last_undegraded":"2024-04-28T12:48:41.639751+0000","last_fullsized":"2024-04-28T12:48:41.639751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:46:03.775693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:08.746870+0000","last_change":"2024-04-28T12:48:33.834069+0000","last_active":"2024-04-28T12:49:08.746870+0000","last_peered":"2024-04-28T12:49:08.746870+0000","last_clean":"2024-04-28T12:49:08.746870+0000","last_became_active":"2024-04-28T12:48:33.833777+0000","last_became_peered":"2024-04-28T12:48:33.833777+0000","last_unstale":"2024-04-28T12:49:08.746870+0000","last_undegraded":"2024-04-28T12:49:08.746870+0000","last_fullsized":"2024-04-28T12:49:08.746870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:32:52.737718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":347,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637848+0000","last_change":"2024-04-28T12:47:13.177589+0000","last_active":"2024-04-28T12:48:41.637848+0000","last_peered":"2024-04-28T12:48:41.637848+0000","last_clean":"2024-04-28T12:48:41.637848+0000","last_became_active":"2024-04-28T12:47:13.177133+0000","last_became_peered":"2024-04-28T12:47:13.177133+0000","last_unstale":"2024-04-28T12:48:41.637848+0000","last_undegraded":"2024-04-28T12:48:41.637848+0000","last_fullsized":"2024-04-28T12:48:41.637848+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:43:15.851674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:43:15.851674+0000","last_clean_scrub_stamp":"2024-04-28T12:43:15.851674+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:23:21.868028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576330+0000","last_change":"2024-04-28T12:48:39.913691+0000","last_active":"2024-04-28T12:48:40.576330+0000","last_peered":"2024-04-28T12:48:40.576330+0000","last_clean":"2024-04-28T12:48:40.576330+0000","last_became_active":"2024-04-28T12:48:39.912989+0000","last_became_peered":"2024-04-28T12:48:39.912989+0000","last_unstale":"2024-04-28T12:48:40.576330+0000","last_undegraded":"2024-04-28T12:48:40.576330+0000","last_fullsized":"2024-04-28T12:48:40.576330+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:49:01.712194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.578039+0000","last_change":"2024-04-28T12:48:34.550976+0000","last_active":"2024-04-28T12:48:40.578039+0000","last_peered":"2024-04-28T12:48:40.578039+0000","last_clean":"2024-04-28T12:48:40.578039+0000","last_became_active":"2024-04-28T12:48:34.549460+0000","last_became_peered":"2024-04-28T12:48:34.549460+0000","last_unstale":"2024-04-28T12:48:40.578039+0000","last_undegraded":"2024-04-28T12:48:40.578039+0000","last_fullsized":"2024-04-28T12:48:40.578039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:55:16.272917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:07.737998+0000","last_change":"2024-04-28T12:48:33.837248+0000","last_active":"2024-04-28T12:49:07.737998+0000","last_peered":"2024-04-28T12:49:07.737998+0000","last_clean":"2024-04-28T12:49:07.737998+0000","last_became_active":"2024-04-28T12:48:33.836910+0000","last_became_peered":"2024-04-28T12:48:33.836910+0000","last_unstale":"2024-04-28T12:49:07.737998+0000","last_undegraded":"2024-04-28T12:49:07.737998+0000","last_fullsized":"2024-04-28T12:49:07.737998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:06:19.572724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639266+0000","last_change":"2024-04-28T12:48:39.605436+0000","last_active":"2024-04-28T12:48:41.639266+0000","last_peered":"2024-04-28T12:48:41.639266+0000","last_clean":"2024-04-28T12:48:41.639266+0000","last_became_active":"2024-04-28T12:48:39.604997+0000","last_became_peered":"2024-04-28T12:48:39.604997+0000","last_unstale":"2024-04-28T12:48:41.639266+0000","last_undegraded":"2024-04-28T12:48:41.639266+0000","last_fullsized":"2024-04-28T12:48:41.639266+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:05:11.571064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638257+0000","last_change":"2024-04-28T12:48:33.848257+0000","last_active":"2024-04-28T12:48:41.638257+0000","last_peered":"2024-04-28T12:48:41.638257+0000","last_clean":"2024-04-28T12:48:41.638257+0000","last_became_active":"2024-04-28T12:48:33.848066+0000","last_became_peered":"2024-04-28T12:48:33.848066+0000","last_unstale":"2024-04-28T12:48:41.638257+0000","last_undegraded":"2024-04-28T12:48:41.638257+0000","last_fullsized":"2024-04-28T12:48:41.638257+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:58:34.174153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579714+0000","last_change":"2024-04-28T12:48:34.559791+0000","last_active":"2024-04-28T12:48:40.579714+0000","last_peered":"2024-04-28T12:48:40.579714+0000","last_clean":"2024-04-28T12:48:40.579714+0000","last_became_active":"2024-04-28T12:48:34.556980+0000","last_became_peered":"2024-04-28T12:48:34.556980+0000","last_unstale":"2024-04-28T12:48:40.579714+0000","last_undegraded":"2024-04-28T12:48:40.579714+0000","last_fullsized":"2024-04-28T12:48:40.579714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:38:56.998159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638933+0000","last_change":"2024-04-28T12:48:39.917423+0000","last_active":"2024-04-28T12:48:41.638933+0000","last_peered":"2024-04-28T12:48:41.638933+0000","last_clean":"2024-04-28T12:48:41.638933+0000","last_became_active":"2024-04-28T12:48:39.914893+0000","last_became_peered":"2024-04-28T12:48:39.914893+0000","last_unstale":"2024-04-28T12:48:41.638933+0000","last_undegraded":"2024-04-28T12:48:41.638933+0000","last_fullsized":"2024-04-28T12:48:41.638933+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:08:05.499808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.524270+0000","last_change":"2024-04-28T12:48:33.841383+0000","last_active":"2024-04-28T12:48:41.524270+0000","last_peered":"2024-04-28T12:48:41.524270+0000","last_clean":"2024-04-28T12:48:41.524270+0000","last_became_active":"2024-04-28T12:48:33.838480+0000","last_became_peered":"2024-04-28T12:48:33.838480+0000","last_unstale":"2024-04-28T12:48:41.524270+0000","last_undegraded":"2024-04-28T12:48:41.524270+0000","last_fullsized":"2024-04-28T12:48:41.524270+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:53:18.795662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.641395+0000","last_change":"2024-04-28T12:48:34.566120+0000","last_active":"2024-04-28T12:48:41.641395+0000","last_peered":"2024-04-28T12:48:41.641395+0000","last_clean":"2024-04-28T12:48:41.641395+0000","last_became_active":"2024-04-28T12:48:34.564357+0000","last_became_peered":"2024-04-28T12:48:34.564357+0000","last_unstale":"2024-04-28T12:48:41.641395+0000","last_undegraded":"2024-04-28T12:48:41.641395+0000","last_fullsized":"2024-04-28T12:48:41.641395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:28:43.463008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638386+0000","last_change":"2024-04-28T12:48:39.586362+0000","last_active":"2024-04-28T12:48:41.638386+0000","last_peered":"2024-04-28T12:48:41.638386+0000","last_clean":"2024-04-28T12:48:41.638386+0000","last_became_active":"2024-04-28T12:48:39.585898+0000","last_became_peered":"2024-04-28T12:48:39.585898+0000","last_unstale":"2024-04-28T12:48:41.638386+0000","last_undegraded":"2024-04-28T12:48:41.638386+0000","last_fullsized":"2024-04-28T12:48:41.638386+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:51:19.098797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.726144+0000","last_change":"2024-04-28T12:48:33.833163+0000","last_active":"2024-04-28T12:49:06.726144+0000","last_peered":"2024-04-28T12:49:06.726144+0000","last_clean":"2024-04-28T12:49:06.726144+0000","last_became_active":"2024-04-28T12:48:33.832798+0000","last_became_peered":"2024-04-28T12:48:33.832798+0000","last_unstale":"2024-04-28T12:49:06.726144+0000","last_undegraded":"2024-04-28T12:49:06.726144+0000","last_fullsized":"2024-04-28T12:49:06.726144+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:42:07.511346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640879+0000","last_change":"2024-04-28T12:48:34.576917+0000","last_active":"2024-04-28T12:48:41.640879+0000","last_peered":"2024-04-28T12:48:41.640879+0000","last_clean":"2024-04-28T12:48:41.640879+0000","last_became_active":"2024-04-28T12:48:34.575602+0000","last_became_peered":"2024-04-28T12:48:34.575602+0000","last_unstale":"2024-04-28T12:48:41.640879+0000","last_undegraded":"2024-04-28T12:48:41.640879+0000","last_fullsized":"2024-04-28T12:48:41.640879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:47:21.769208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639205+0000","last_change":"2024-04-28T12:48:39.913164+0000","last_active":"2024-04-28T12:48:41.639205+0000","last_peered":"2024-04-28T12:48:41.639205+0000","last_clean":"2024-04-28T12:48:41.639205+0000","last_became_active":"2024-04-28T12:48:39.912716+0000","last_became_peered":"2024-04-28T12:48:39.912716+0000","last_unstale":"2024-04-28T12:48:41.639205+0000","last_undegraded":"2024-04-28T12:48:41.639205+0000","last_fullsized":"2024-04-28T12:48:41.639205+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:26:34.544275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.728395+0000","last_change":"2024-04-28T12:48:33.831286+0000","last_active":"2024-04-28T12:49:06.728395+0000","last_peered":"2024-04-28T12:49:06.728395+0000","last_clean":"2024-04-28T12:49:06.728395+0000","last_became_active":"2024-04-28T12:48:33.830846+0000","last_became_peered":"2024-04-28T12:48:33.830846+0000","last_unstale":"2024-04-28T12:49:06.728395+0000","last_undegraded":"2024-04-28T12:49:06.728395+0000","last_fullsized":"2024-04-28T12:49:06.728395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:32:47.985987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637653+0000","last_change":"2024-04-28T12:48:34.543780+0000","last_active":"2024-04-28T12:48:41.637653+0000","last_peered":"2024-04-28T12:48:41.637653+0000","last_clean":"2024-04-28T12:48:41.637653+0000","last_became_active":"2024-04-28T12:48:34.543441+0000","last_became_peered":"2024-04-28T12:48:34.543441+0000","last_unstale":"2024-04-28T12:48:41.637653+0000","last_undegraded":"2024-04-28T12:48:41.637653+0000","last_fullsized":"2024-04-28T12:48:41.637653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:05:43.334523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.524440+0000","last_change":"2024-04-28T12:48:39.607360+0000","last_active":"2024-04-28T12:48:41.524440+0000","last_peered":"2024-04-28T12:48:41.524440+0000","last_clean":"2024-04-28T12:48:41.524440+0000","last_became_active":"2024-04-28T12:48:39.606944+0000","last_became_peered":"2024-04-28T12:48:39.606944+0000","last_unstale":"2024-04-28T12:48:41.524440+0000","last_undegraded":"2024-04-28T12:48:41.524440+0000","last_fullsized":"2024-04-28T12:48:41.524440+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:59:15.020193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.729166+0000","last_change":"2024-04-28T12:48:33.835694+0000","last_active":"2024-04-28T12:49:06.729166+0000","last_peered":"2024-04-28T12:49:06.729166+0000","last_clean":"2024-04-28T12:49:06.729166+0000","last_became_active":"2024-04-28T12:48:33.835210+0000","last_became_peered":"2024-04-28T12:48:33.835210+0000","last_unstale":"2024-04-28T12:49:06.729166+0000","last_undegraded":"2024-04-28T12:49:06.729166+0000","last_fullsized":"2024-04-28T12:49:06.729166+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:22:34.657665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638809+0000","last_change":"2024-04-28T12:48:34.556933+0000","last_active":"2024-04-28T12:48:41.638809+0000","last_peered":"2024-04-28T12:48:41.638809+0000","last_clean":"2024-04-28T12:48:41.638809+0000","last_became_active":"2024-04-28T12:48:34.556628+0000","last_became_peered":"2024-04-28T12:48:34.556628+0000","last_unstale":"2024-04-28T12:48:41.638809+0000","last_undegraded":"2024-04-28T12:48:41.638809+0000","last_fullsized":"2024-04-28T12:48:41.638809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:40:50.464723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638829+0000","last_change":"2024-04-28T12:48:39.594808+0000","last_active":"2024-04-28T12:48:41.638829+0000","last_peered":"2024-04-28T12:48:41.638829+0000","last_clean":"2024-04-28T12:48:41.638829+0000","last_became_active":"2024-04-28T12:48:39.594333+0000","last_became_peered":"2024-04-28T12:48:39.594333+0000","last_unstale":"2024-04-28T12:48:41.638829+0000","last_undegraded":"2024-04-28T12:48:41.638829+0000","last_fullsized":"2024-04-28T12:48:41.638829+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:34:51.548490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638142+0000","last_change":"2024-04-28T12:48:33.836878+0000","last_active":"2024-04-28T12:48:41.638142+0000","last_peered":"2024-04-28T12:48:41.638142+0000","last_clean":"2024-04-28T12:48:41.638142+0000","last_became_active":"2024-04-28T12:48:33.836536+0000","last_became_peered":"2024-04-28T12:48:33.836536+0000","last_unstale":"2024-04-28T12:48:41.638142+0000","last_undegraded":"2024-04-28T12:48:41.638142+0000","last_fullsized":"2024-04-28T12:48:41.638142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:50:41.064854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640549+0000","last_change":"2024-04-28T12:48:34.575213+0000","last_active":"2024-04-28T12:48:41.640549+0000","last_peered":"2024-04-28T12:48:41.640549+0000","last_clean":"2024-04-28T12:48:41.640549+0000","last_became_active":"2024-04-28T12:48:34.574892+0000","last_became_peered":"2024-04-28T12:48:34.574892+0000","last_unstale":"2024-04-28T12:48:41.640549+0000","last_undegraded":"2024-04-28T12:48:41.640549+0000","last_fullsized":"2024-04-28T12:48:41.640549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:27:54.144215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639384+0000","last_change":"2024-04-28T12:48:39.912466+0000","last_active":"2024-04-28T12:48:41.639384+0000","last_peered":"2024-04-28T12:48:41.639384+0000","last_clean":"2024-04-28T12:48:41.639384+0000","last_became_active":"2024-04-28T12:48:39.911914+0000","last_became_peered":"2024-04-28T12:48:39.911914+0000","last_unstale":"2024-04-28T12:48:41.639384+0000","last_undegraded":"2024-04-28T12:48:41.639384+0000","last_fullsized":"2024-04-28T12:48:41.639384+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:26:35.341389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:06.726982+0000","last_change":"2024-04-28T12:48:33.840024+0000","last_active":"2024-04-28T12:49:06.726982+0000","last_peered":"2024-04-28T12:49:06.726982+0000","last_clean":"2024-04-28T12:49:06.726982+0000","last_became_active":"2024-04-28T12:48:33.838608+0000","last_became_peered":"2024-04-28T12:48:33.838608+0000","last_unstale":"2024-04-28T12:49:06.726982+0000","last_undegraded":"2024-04-28T12:49:06.726982+0000","last_fullsized":"2024-04-28T12:49:06.726982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:31:28.060385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638864+0000","last_change":"2024-04-28T12:48:34.554685+0000","last_active":"2024-04-28T12:48:41.638864+0000","last_peered":"2024-04-28T12:48:41.638864+0000","last_clean":"2024-04-28T12:48:41.638864+0000","last_became_active":"2024-04-28T12:48:34.554305+0000","last_became_peered":"2024-04-28T12:48:34.554305+0000","last_unstale":"2024-04-28T12:48:41.638864+0000","last_undegraded":"2024-04-28T12:48:41.638864+0000","last_fullsized":"2024-04-28T12:48:41.638864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:19:53.698623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576384+0000","last_change":"2024-04-28T12:48:39.605409+0000","last_active":"2024-04-28T12:48:40.576384+0000","last_peered":"2024-04-28T12:48:40.576384+0000","last_clean":"2024-04-28T12:48:40.576384+0000","last_became_active":"2024-04-28T12:48:39.604991+0000","last_became_peered":"2024-04-28T12:48:39.604991+0000","last_unstale":"2024-04-28T12:48:40.576384+0000","last_undegraded":"2024-04-28T12:48:40.576384+0000","last_fullsized":"2024-04-28T12:48:40.576384+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:37:44.481211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:49:07.741115+0000","last_change":"2024-04-28T12:48:33.835286+0000","last_active":"2024-04-28T12:49:07.741115+0000","last_peered":"2024-04-28T12:49:07.741115+0000","last_clean":"2024-04-28T12:49:07.741115+0000","last_became_active":"2024-04-28T12:48:33.834729+0000","last_became_peered":"2024-04-28T12:48:33.834729+0000","last_unstale":"2024-04-28T12:49:07.741115+0000","last_undegraded":"2024-04-28T12:49:07.741115+0000","last_fullsized":"2024-04-28T12:49:07.741115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:52:54.889021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639197+0000","last_change":"2024-04-28T12:48:34.547865+0000","last_active":"2024-04-28T12:48:41.639197+0000","last_peered":"2024-04-28T12:48:41.639197+0000","last_clean":"2024-04-28T12:48:41.639197+0000","last_became_active":"2024-04-28T12:48:34.547226+0000","last_became_peered":"2024-04-28T12:48:34.547226+0000","last_unstale":"2024-04-28T12:48:41.639197+0000","last_undegraded":"2024-04-28T12:48:41.639197+0000","last_fullsized":"2024-04-28T12:48:41.639197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:14:06.376908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640100+0000","last_change":"2024-04-28T12:48:39.611010+0000","last_active":"2024-04-28T12:48:41.640100+0000","last_peered":"2024-04-28T12:48:41.640100+0000","last_clean":"2024-04-28T12:48:41.640100+0000","last_became_active":"2024-04-28T12:48:39.610475+0000","last_became_peered":"2024-04-28T12:48:39.610475+0000","last_unstale":"2024-04-28T12:48:41.640100+0000","last_undegraded":"2024-04-28T12:48:41.640100+0000","last_fullsized":"2024-04-28T12:48:41.640100+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:19:44.707837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576717+0000","last_change":"2024-04-28T12:48:33.835109+0000","last_active":"2024-04-28T12:48:40.576717+0000","last_peered":"2024-04-28T12:48:40.576717+0000","last_clean":"2024-04-28T12:48:40.576717+0000","last_became_active":"2024-04-28T12:48:33.834757+0000","last_became_peered":"2024-04-28T12:48:33.834757+0000","last_unstale":"2024-04-28T12:48:40.576717+0000","last_undegraded":"2024-04-28T12:48:40.576717+0000","last_fullsized":"2024-04-28T12:48:40.576717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:46:45.884261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638235+0000","last_change":"2024-04-28T12:48:34.565135+0000","last_active":"2024-04-28T12:48:41.638235+0000","last_peered":"2024-04-28T12:48:41.638235+0000","last_clean":"2024-04-28T12:48:41.638235+0000","last_became_active":"2024-04-28T12:48:34.564631+0000","last_became_peered":"2024-04-28T12:48:34.564631+0000","last_unstale":"2024-04-28T12:48:41.638235+0000","last_undegraded":"2024-04-28T12:48:41.638235+0000","last_fullsized":"2024-04-28T12:48:41.638235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:36:46.747085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.575869+0000","last_change":"2024-04-28T12:48:39.591530+0000","last_active":"2024-04-28T12:48:40.575869+0000","last_peered":"2024-04-28T12:48:40.575869+0000","last_clean":"2024-04-28T12:48:40.575869+0000","last_became_active":"2024-04-28T12:48:39.591016+0000","last_became_peered":"2024-04-28T12:48:39.591016+0000","last_unstale":"2024-04-28T12:48:40.575869+0000","last_undegraded":"2024-04-28T12:48:40.575869+0000","last_fullsized":"2024-04-28T12:48:40.575869+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:04:32.281117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579649+0000","last_change":"2024-04-28T12:48:33.835565+0000","last_active":"2024-04-28T12:48:40.579649+0000","last_peered":"2024-04-28T12:48:40.579649+0000","last_clean":"2024-04-28T12:48:40.579649+0000","last_became_active":"2024-04-28T12:48:33.835147+0000","last_became_peered":"2024-04-28T12:48:33.835147+0000","last_unstale":"2024-04-28T12:48:40.579649+0000","last_undegraded":"2024-04-28T12:48:40.579649+0000","last_fullsized":"2024-04-28T12:48:40.579649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:48:23.520295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639449+0000","last_change":"2024-04-28T12:48:34.547561+0000","last_active":"2024-04-28T12:48:41.639449+0000","last_peered":"2024-04-28T12:48:41.639449+0000","last_clean":"2024-04-28T12:48:41.639449+0000","last_became_active":"2024-04-28T12:48:34.547236+0000","last_became_peered":"2024-04-28T12:48:34.547236+0000","last_unstale":"2024-04-28T12:48:41.639449+0000","last_undegraded":"2024-04-28T12:48:41.639449+0000","last_fullsized":"2024-04-28T12:48:41.639449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:00:37.150080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639007+0000","last_change":"2024-04-28T12:48:34.546027+0000","last_active":"2024-04-28T12:48:41.639007+0000","last_peered":"2024-04-28T12:48:41.639007+0000","last_clean":"2024-04-28T12:48:41.639007+0000","last_became_active":"2024-04-28T12:48:34.545618+0000","last_became_peered":"2024-04-28T12:48:34.545618+0000","last_unstale":"2024-04-28T12:48:41.639007+0000","last_undegraded":"2024-04-28T12:48:41.639007+0000","last_fullsized":"2024-04-28T12:48:41.639007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:40:00.928586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638680+0000","last_change":"2024-04-28T12:48:34.544883+0000","last_active":"2024-04-28T12:48:41.638680+0000","last_peered":"2024-04-28T12:48:41.638680+0000","last_clean":"2024-04-28T12:48:41.638680+0000","last_became_active":"2024-04-28T12:48:34.544454+0000","last_became_peered":"2024-04-28T12:48:34.544454+0000","last_unstale":"2024-04-28T12:48:41.638680+0000","last_undegraded":"2024-04-28T12:48:41.638680+0000","last_fullsized":"2024-04-28T12:48:41.638680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:58:16.064432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.637981+0000","last_change":"2024-04-28T12:48:34.565110+0000","last_active":"2024-04-28T12:48:41.637981+0000","last_peered":"2024-04-28T12:48:41.637981+0000","last_clean":"2024-04-28T12:48:41.637981+0000","last_became_active":"2024-04-28T12:48:34.564723+0000","last_became_peered":"2024-04-28T12:48:34.564723+0000","last_unstale":"2024-04-28T12:48:41.637981+0000","last_undegraded":"2024-04-28T12:48:41.637981+0000","last_fullsized":"2024-04-28T12:48:41.637981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:58:24.220515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.640384+0000","last_change":"2024-04-28T12:48:34.572428+0000","last_active":"2024-04-28T12:48:41.640384+0000","last_peered":"2024-04-28T12:48:41.640384+0000","last_clean":"2024-04-28T12:48:41.640384+0000","last_became_active":"2024-04-28T12:48:34.572034+0000","last_became_peered":"2024-04-28T12:48:34.572034+0000","last_unstale":"2024-04-28T12:48:41.640384+0000","last_undegraded":"2024-04-28T12:48:41.640384+0000","last_fullsized":"2024-04-28T12:48:41.640384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:38:23.598568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579659+0000","last_change":"2024-04-28T12:48:34.562425+0000","last_active":"2024-04-28T12:48:40.579659+0000","last_peered":"2024-04-28T12:48:40.579659+0000","last_clean":"2024-04-28T12:48:40.579659+0000","last_became_active":"2024-04-28T12:48:34.562087+0000","last_became_peered":"2024-04-28T12:48:34.562087+0000","last_unstale":"2024-04-28T12:48:40.579659+0000","last_undegraded":"2024-04-28T12:48:40.579659+0000","last_fullsized":"2024-04-28T12:48:40.579659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:32:39.701796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639546+0000","last_change":"2024-04-28T12:48:34.547755+0000","last_active":"2024-04-28T12:48:41.639546+0000","last_peered":"2024-04-28T12:48:41.639546+0000","last_clean":"2024-04-28T12:48:41.639546+0000","last_became_active":"2024-04-28T12:48:34.547427+0000","last_became_peered":"2024-04-28T12:48:34.547427+0000","last_unstale":"2024-04-28T12:48:41.639546+0000","last_undegraded":"2024-04-28T12:48:41.639546+0000","last_fullsized":"2024-04-28T12:48:41.639546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:58:15.694252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.638706+0000","last_change":"2024-04-28T12:48:34.543146+0000","last_active":"2024-04-28T12:48:41.638706+0000","last_peered":"2024-04-28T12:48:41.638706+0000","last_clean":"2024-04-28T12:48:41.638706+0000","last_became_active":"2024-04-28T12:48:34.542809+0000","last_became_peered":"2024-04-28T12:48:34.542809+0000","last_unstale":"2024-04-28T12:48:41.638706+0000","last_undegraded":"2024-04-28T12:48:41.638706+0000","last_fullsized":"2024-04-28T12:48:41.638706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:09:42.515358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.641607+0000","last_change":"2024-04-28T12:48:34.571166+0000","last_active":"2024-04-28T12:48:41.641607+0000","last_peered":"2024-04-28T12:48:41.641607+0000","last_clean":"2024-04-28T12:48:41.641607+0000","last_became_active":"2024-04-28T12:48:34.567803+0000","last_became_peered":"2024-04-28T12:48:34.567803+0000","last_unstale":"2024-04-28T12:48:41.641607+0000","last_undegraded":"2024-04-28T12:48:41.641607+0000","last_fullsized":"2024-04-28T12:48:41.641607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:11:09.945751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576353+0000","last_change":"2024-04-28T12:48:34.564939+0000","last_active":"2024-04-28T12:48:40.576353+0000","last_peered":"2024-04-28T12:48:40.576353+0000","last_clean":"2024-04-28T12:48:40.576353+0000","last_became_active":"2024-04-28T12:48:34.564486+0000","last_became_peered":"2024-04-28T12:48:34.564486+0000","last_unstale":"2024-04-28T12:48:40.576353+0000","last_undegraded":"2024-04-28T12:48:40.576353+0000","last_fullsized":"2024-04-28T12:48:40.576353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:29:44.618469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639204+0000","last_change":"2024-04-28T12:48:34.567359+0000","last_active":"2024-04-28T12:48:41.639204+0000","last_peered":"2024-04-28T12:48:41.639204+0000","last_clean":"2024-04-28T12:48:41.639204+0000","last_became_active":"2024-04-28T12:48:34.565393+0000","last_became_peered":"2024-04-28T12:48:34.565393+0000","last_unstale":"2024-04-28T12:48:41.639204+0000","last_undegraded":"2024-04-28T12:48:41.639204+0000","last_fullsized":"2024-04-28T12:48:41.639204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:44:04.312257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.576675+0000","last_change":"2024-04-28T12:48:34.565019+0000","last_active":"2024-04-28T12:48:40.576675+0000","last_peered":"2024-04-28T12:48:40.576675+0000","last_clean":"2024-04-28T12:48:40.576675+0000","last_became_active":"2024-04-28T12:48:34.564635+0000","last_became_peered":"2024-04-28T12:48:34.564635+0000","last_unstale":"2024-04-28T12:48:40.576675+0000","last_undegraded":"2024-04-28T12:48:40.576675+0000","last_fullsized":"2024-04-28T12:48:40.576675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:20.645281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:40.579260+0000","last_change":"2024-04-28T12:48:34.574043+0000","last_active":"2024-04-28T12:48:40.579260+0000","last_peered":"2024-04-28T12:48:40.579260+0000","last_clean":"2024-04-28T12:48:40.579260+0000","last_became_active":"2024-04-28T12:48:34.573686+0000","last_became_peered":"2024-04-28T12:48:34.573686+0000","last_unstale":"2024-04-28T12:48:40.579260+0000","last_undegraded":"2024-04-28T12:48:40.579260+0000","last_fullsized":"2024-04-28T12:48:40.579260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:29:57.371318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-28T12:48:41.639294+0000","last_change":"2024-04-28T12:48:34.547404+0000","last_active":"2024-04-28T12:48:41.639294+0000","last_peered":"2024-04-28T12:48:41.639294+0000","last_clean":"2024-04-28T12:48:41.639294+0000","last_became_active":"2024-04-28T12:48:34.546581+0000","last_became_peered":"2024-04-28T12:48:34.546581+0000","last_unstale":"2024-04-28T12:48:41.639294+0000","last_undegraded":"2024-04-28T12:48:41.639294+0000","last_fullsized":"2024-04-28T12:48:41.639294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:43:21.389455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281028,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14722,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":23,"num_read_kb":22,"num_write":118,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":86502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":485394,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645337,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1081182,"data_compressed":50466,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808864,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":486178,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070501,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":485998,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332140,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1086528,"data_compressed":50466,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495665,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":485484,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":525865,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":486872,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":490586,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182467,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":485678,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":486429,"data_compressed":14721,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":1077238,"data_compressed":50466,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34101,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34101,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34101,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":40471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":24882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-28T12:49:13.112 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-28T12:49:13.112 INFO:tasks.cephadm:Setting up client nodes... 2024-04-28T12:49:13.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-28T12:49:13.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:12 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3549018296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:13.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:12 smithi179 ceph-mon[63029]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:49:13.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:12 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3549018296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:13.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:12 smithi161 ceph-mon[62387]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:49:13.600 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:14.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:13 smithi084 ceph-mon[28081]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:49:14.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:13 smithi084 ceph-mon[28081]: pgmap v342: 129 pgs: 129 active+clean; 591 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-04-28T12:49:14.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:13 smithi179 ceph-mon[63029]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:49:14.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:13 smithi179 ceph-mon[63029]: pgmap v342: 129 pgs: 129 active+clean; 591 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-04-28T12:49:14.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:13 smithi161 ceph-mon[62387]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:49:14.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:13 smithi161 ceph-mon[62387]: pgmap v342: 129 pgs: 129 active+clean; 591 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-04-28T12:49:16.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.151+0000 7fe1e2e00700 1 -- 172.21.15.84:0/1327538926 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1dc1068d0 msgr2=0x7fe1dc106d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:16.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.151+0000 7fe1e2e00700 1 --2- 172.21.15.84:0/1327538926 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1dc1068d0 0x7fe1dc106d30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80099c0 tx=0x7fe1c8009cd0 comp rx=0 tx=0).stop 2024-04-28T12:49:16.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.151+0000 7fe1e2e00700 1 -- 172.21.15.84:0/1327538926 shutdown_connections 2024-04-28T12:49:16.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.151+0000 7fe1e2e00700 1 --2- 172.21.15.84:0/1327538926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1dc107270 0x7fe1dc10db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:16.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.151+0000 7fe1e2e00700 1 --2- 172.21.15.84:0/1327538926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1dc1056d0 0x7fe1dc105af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:16.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.151+0000 7fe1e2e00700 1 --2- 172.21.15.84:0/1327538926 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1dc1068d0 0x7fe1dc106d30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:16.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.151+0000 7fe1e2e00700 1 -- 172.21.15.84:0/1327538926 >> 172.21.15.84:0/1327538926 conn(0x7fe1dc100c70 msgr2=0x7fe1dc1030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:16.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.151+0000 7fe1e2e00700 1 -- 172.21.15.84:0/1327538926 shutdown_connections 2024-04-28T12:49:16.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.151+0000 7fe1e2e00700 1 -- 172.21.15.84:0/1327538926 wait complete. 2024-04-28T12:49:16.148 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.152+0000 7fe1e2e00700 1 Processor -- start 2024-04-28T12:49:16.149 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.152+0000 7fe1e2e00700 1 -- start start 2024-04-28T12:49:16.150 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.153+0000 7fe1e2e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1dc1056d0 0x7fe1dc19b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:16.150 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.153+0000 7fe1e2e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1dc1068d0 0x7fe1dc19bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:16.150 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.153+0000 7fe1e2e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1dc107270 0x7fe1dc1a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:16.150 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.153+0000 7fe1e2e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc1111b0 con 0x7fe1dc1068d0 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.153+0000 7fe1e2e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc111030 con 0x7fe1dc107270 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.153+0000 7fe1e2e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe1dc111330 con 0x7fe1dc1056d0 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.153+0000 7fe1db400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1dc1068d0 0x7fe1dc19bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.153+0000 7fe1db400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1dc1068d0 0x7fe1dc19bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43294/0 (socket says 172.21.15.84:43294) 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.153+0000 7fe1dbe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1dc1056d0 0x7fe1dc19b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.153+0000 7fe1db400700 1 -- 172.21.15.84:0/2852296089 learned_addr learned my addr 172.21.15.84:0/2852296089 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.153+0000 7fe1e0e00700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1dc107270 0x7fe1dc1a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.154+0000 7fe1db400700 1 -- 172.21.15.84:0/2852296089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1dc1056d0 msgr2=0x7fe1dc19b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.154+0000 7fe1db400700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1dc1056d0 0x7fe1dc19b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.154+0000 7fe1db400700 1 -- 172.21.15.84:0/2852296089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1dc107270 msgr2=0x7fe1dc1a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.154+0000 7fe1db400700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1dc107270 0x7fe1dc1a2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.154+0000 7fe1db400700 1 -- 172.21.15.84:0/2852296089 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1dc1a33c0 con 0x7fe1dc1068d0 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.154+0000 7fe1dbe00700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1dc1056d0 0x7fe1dc19b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.154+0000 7fe1db400700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1dc1068d0 0x7fe1dc19bca0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8005910 tx=0x7fe1c8003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:16.151 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.154+0000 7fe1d8c00700 1 -- 172.21.15.84:0/2852296089 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8009e60 con 0x7fe1dc1068d0 2024-04-28T12:49:16.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.154+0000 7fe1d8c00700 1 -- 172.21.15.84:0/2852296089 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1c800f460 con 0x7fe1dc1068d0 2024-04-28T12:49:16.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.154+0000 7fe1e2e00700 1 -- 172.21.15.84:0/2852296089 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1dc1a3650 con 0x7fe1dc1068d0 2024-04-28T12:49:16.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.154+0000 7fe1d8c00700 1 -- 172.21.15.84:0/2852296089 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8017650 con 0x7fe1dc1068d0 2024-04-28T12:49:16.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.155+0000 7fe1e2e00700 1 -- 172.21.15.84:0/2852296089 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1dc1a3aa0 con 0x7fe1dc1068d0 2024-04-28T12:49:16.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.156+0000 7fe1e2e00700 1 -- 172.21.15.84:0/2852296089 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1dc051960 con 0x7fe1dc1068d0 2024-04-28T12:49:16.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.159+0000 7fe1d8c00700 1 -- 172.21.15.84:0/2852296089 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe1c800f910 con 0x7fe1dc1068d0 2024-04-28T12:49:16.155 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.160+0000 7fe1d8c00700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe1c00a3d80 0x7fe1c00a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:16.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.160+0000 7fe1d8c00700 1 -- 172.21.15.84:0/2852296089 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe1c80d1710 con 0x7fe1dc1068d0 2024-04-28T12:49:16.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.161+0000 7fe1dbe00700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe1c00a3d80 0x7fe1c00a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:16.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.161+0000 7fe1dbe00700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe1c00a3d80 0x7fe1c00a6240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc106730 tx=0x7fe1cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:16.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.162+0000 7fe1d8c00700 1 -- 172.21.15.84:0/2852296089 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe1c8013070 con 0x7fe1dc1068d0 2024-04-28T12:49:16.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.479+0000 7fe1e2e00700 1 -- 172.21.15.84:0/2852296089 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fe1dc02e4c0 con 0x7fe1dc1068d0 2024-04-28T12:49:16.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.482+0000 7fe1d8c00700 1 -- 172.21.15.84:0/2852296089 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fe1c809dd80 con 0x7fe1dc1068d0 2024-04-28T12:49:16.477 INFO:teuthology.orchestra.run.smithi084.stdout:[client.0] 2024-04-28T12:49:16.478 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQDMRS5mpoijHBAACiP9b+3VJPHZrfd/C1aZZQ== 2024-04-28T12:49:16.481 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.485+0000 7fe1e2e00700 1 -- 172.21.15.84:0/2852296089 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe1c00a3d80 msgr2=0x7fe1c00a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:16.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.485+0000 7fe1e2e00700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe1c00a3d80 0x7fe1c00a6240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc106730 tx=0x7fe1cc009260 comp rx=0 tx=0).stop 2024-04-28T12:49:16.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.485+0000 7fe1e2e00700 1 -- 172.21.15.84:0/2852296089 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1dc1068d0 msgr2=0x7fe1dc19bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:16.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.485+0000 7fe1e2e00700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1dc1068d0 0x7fe1dc19bca0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8005910 tx=0x7fe1c8003e60 comp rx=0 tx=0).stop 2024-04-28T12:49:16.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.486+0000 7fe1e2e00700 1 -- 172.21.15.84:0/2852296089 shutdown_connections 2024-04-28T12:49:16.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.486+0000 7fe1e2e00700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe1c00a3d80 0x7fe1c00a6240 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:16.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.486+0000 7fe1e2e00700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1dc107270 0x7fe1dc1a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:16.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.486+0000 7fe1e2e00700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1dc1056d0 0x7fe1dc19b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:16.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.486+0000 7fe1e2e00700 1 --2- 172.21.15.84:0/2852296089 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1dc1068d0 0x7fe1dc19bca0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:16.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.486+0000 7fe1e2e00700 1 -- 172.21.15.84:0/2852296089 >> 172.21.15.84:0/2852296089 conn(0x7fe1dc100c70 msgr2=0x7fe1dc109b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:16.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.486+0000 7fe1e2e00700 1 -- 172.21.15.84:0/2852296089 shutdown_connections 2024-04-28T12:49:16.482 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:16.486+0000 7fe1e2e00700 1 -- 172.21.15.84:0/2852296089 wait complete. 2024-04-28T12:49:16.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:16 smithi084 ceph-mon[28081]: pgmap v343: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 13 op/s 2024-04-28T12:49:16.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:16 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2852296089' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-28T12:49:16.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:16 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2852296089' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-28T12:49:16.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:16 smithi161 ceph-mon[62387]: pgmap v343: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 13 op/s 2024-04-28T12:49:16.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:16 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2852296089' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-28T12:49:16.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:16 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2852296089' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-28T12:49:17.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:16 smithi179 ceph-mon[63029]: pgmap v343: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 13 op/s 2024-04-28T12:49:17.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:16 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2852296089' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-28T12:49:17.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:16 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2852296089' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-28T12:49:17.347 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:49:17.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-28T12:49:17.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-28T12:49:17.485 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-28T12:49:17.485 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-28T12:49:17.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-28T12:49:17.971 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:18.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:18 smithi161 ceph-mon[62387]: pgmap v344: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 13 op/s 2024-04-28T12:49:19.081 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:18 smithi084 ceph-mon[28081]: pgmap v344: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 13 op/s 2024-04-28T12:49:19.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:18 smithi179 ceph-mon[63029]: pgmap v344: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 13 op/s 2024-04-28T12:49:20.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:20 smithi084 ceph-mon[28081]: pgmap v345: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 13 op/s 2024-04-28T12:49:20.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.391+0000 7fccca600700 1 -- 172.21.15.84:0/1272483963 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccc4075f70 msgr2=0x7fccc410f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:20.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.391+0000 7fccca600700 1 --2- 172.21.15.84:0/1272483963 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccc4075f70 0x7fccc410f910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009990 tx=0x7fccb0009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:20.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.391+0000 7fccca600700 1 -- 172.21.15.84:0/1272483963 shutdown_connections 2024-04-28T12:49:20.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.391+0000 7fccca600700 1 --2- 172.21.15.84:0/1272483963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccc4075650 0x7fccc4075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:20.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.391+0000 7fccca600700 1 --2- 172.21.15.84:0/1272483963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc410fe50 0x7fccc4112240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:20.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.391+0000 7fccca600700 1 --2- 172.21.15.84:0/1272483963 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccc4075f70 0x7fccc410f910 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:20.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.391+0000 7fccca600700 1 -- 172.21.15.84:0/1272483963 >> 172.21.15.84:0/1272483963 conn(0x7fccc407a4f0 msgr2=0x7fccc407a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:20.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.391+0000 7fccca600700 1 -- 172.21.15.84:0/1272483963 shutdown_connections 2024-04-28T12:49:20.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.391+0000 7fccca600700 1 -- 172.21.15.84:0/1272483963 wait complete. 2024-04-28T12:49:20.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.392+0000 7fccca600700 1 Processor -- start 2024-04-28T12:49:20.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.392+0000 7fccca600700 1 -- start start 2024-04-28T12:49:20.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.392+0000 7fccca600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccc4075650 0x7fccc41a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:20.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.392+0000 7fccca600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccc4075f70 0x7fccc41a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:20.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccca600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc410fe50 0x7fccc419bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:20.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccca600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fccc4115860 con 0x7fccc410fe50 2024-04-28T12:49:20.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccca600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41156e0 con 0x7fccc4075650 2024-04-28T12:49:20.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccca600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41159e0 con 0x7fccc4075f70 2024-04-28T12:49:20.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccc3400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccc4075650 0x7fccc41a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:20.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccc3e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc410fe50 0x7fccc419bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:20.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccc3400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccc4075650 0x7fccc41a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:48596/0 (socket says 172.21.15.84:48596) 2024-04-28T12:49:20.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccc3400700 1 -- 172.21.15.84:0/625189571 learned_addr learned my addr 172.21.15.84:0/625189571 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:20.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccc2a00700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccc4075f70 0x7fccc41a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:20.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccc3e00700 1 -- 172.21.15.84:0/625189571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccc4075f70 msgr2=0x7fccc41a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:20.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccc3e00700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccc4075f70 0x7fccc41a1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:20.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccc3e00700 1 -- 172.21.15.84:0/625189571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccc4075650 msgr2=0x7fccc41a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:20.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccc3e00700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccc4075650 0x7fccc41a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:20.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.393+0000 7fccc3e00700 1 -- 172.21.15.84:0/625189571 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc419c440 con 0x7fccc410fe50 2024-04-28T12:49:20.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.394+0000 7fccc3e00700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc410fe50 0x7fccc419bb60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00ef90 tx=0x7fccbc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:20.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.394+0000 7fccbbe00700 1 -- 172.21.15.84:0/625189571 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc0092e0 con 0x7fccc410fe50 2024-04-28T12:49:20.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.394+0000 7fccbbe00700 1 -- 172.21.15.84:0/625189571 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccbc004bb0 con 0x7fccc410fe50 2024-04-28T12:49:20.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.394+0000 7fccca600700 1 -- 172.21.15.84:0/625189571 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc419c6d0 con 0x7fccc410fe50 2024-04-28T12:49:20.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.394+0000 7fccbbe00700 1 -- 172.21.15.84:0/625189571 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc004020 con 0x7fccc410fe50 2024-04-28T12:49:20.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.394+0000 7fccca600700 1 -- 172.21.15.84:0/625189571 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc41a87f0 con 0x7fccc410fe50 2024-04-28T12:49:20.391 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.395+0000 7fccca600700 1 -- 172.21.15.84:0/625189571 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccc4076d40 con 0x7fccc410fe50 2024-04-28T12:49:20.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.399+0000 7fccbbe00700 1 -- 172.21.15.84:0/625189571 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fccbc01b030 con 0x7fccc410fe50 2024-04-28T12:49:20.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.400+0000 7fccbbe00700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcca80a3cd0 0x7fcca80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:20.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.401+0000 7fccbbe00700 1 -- 172.21.15.84:0/625189571 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fccbc0d0d30 con 0x7fccc410fe50 2024-04-28T12:49:20.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.401+0000 7fccc3400700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcca80a3cd0 0x7fcca80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:20.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.402+0000 7fccc3400700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcca80a3cd0 0x7fcca80a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009680 tx=0x7fccb4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:20.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.403+0000 7fccbbe00700 1 -- 172.21.15.84:0/625189571 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fccbc022020 con 0x7fccc410fe50 2024-04-28T12:49:20.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:20 smithi161 ceph-mon[62387]: pgmap v345: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 13 op/s 2024-04-28T12:49:20.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:20 smithi179 ceph-mon[63029]: pgmap v345: 129 pgs: 129 active+clean; 626 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 13 op/s 2024-04-28T12:49:20.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.699+0000 7fccca600700 1 -- 172.21.15.84:0/625189571 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fccc402e490 con 0x7fccc410fe50 2024-04-28T12:49:20.699 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.704+0000 7fccbbe00700 1 -- 172.21.15.84:0/625189571 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264993 (secure 0 0 0) 0x7fccbc010040 con 0x7fccc410fe50 2024-04-28T12:49:20.701 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:49:20.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.710+0000 7fccca600700 1 -- 172.21.15.84:0/625189571 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcca80a3cd0 msgr2=0x7fcca80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:20.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.710+0000 7fccca600700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcca80a3cd0 0x7fcca80a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009680 tx=0x7fccb4009260 comp rx=0 tx=0).stop 2024-04-28T12:49:20.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.710+0000 7fccca600700 1 -- 172.21.15.84:0/625189571 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc410fe50 msgr2=0x7fccc419bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:20.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.710+0000 7fccca600700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc410fe50 0x7fccc419bb60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00ef90 tx=0x7fccbc00c3a0 comp rx=0 tx=0).stop 2024-04-28T12:49:20.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.710+0000 7fccca600700 1 -- 172.21.15.84:0/625189571 shutdown_connections 2024-04-28T12:49:20.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.710+0000 7fccca600700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcca80a3cd0 0x7fcca80a6190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:20.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.710+0000 7fccca600700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fccc4075650 0x7fccc41a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:20.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.710+0000 7fccca600700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc410fe50 0x7fccc419bb60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:20.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.710+0000 7fccca600700 1 --2- 172.21.15.84:0/625189571 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fccc4075f70 0x7fccc41a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:20.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.710+0000 7fccca600700 1 -- 172.21.15.84:0/625189571 >> 172.21.15.84:0/625189571 conn(0x7fccc407a4f0 msgr2=0x7fccc4100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:20.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.710+0000 7fccca600700 1 -- 172.21.15.84:0/625189571 shutdown_connections 2024-04-28T12:49:20.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:20.710+0000 7fccca600700 1 -- 172.21.15.84:0/625189571 wait complete. 2024-04-28T12:49:21.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:21 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/625189571' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-28T12:49:21.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:21 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/625189571' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-28T12:49:21.516 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":2396287759},{"type":"v1","addr":"172.21.15.84:6801","nonce":2396287759}]},"active_addr":"172.21.15.84:6801/2396287759","active_change":"2024-04-28T12:40:29.445985+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24115,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.84:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":634025401}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":636729980}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":38109523}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":176100185}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":1685774249}]}]} 2024-04-28T12:49:21.520 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-28T12:49:21.520 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-28T12:49:21.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-28T12:49:21.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:21 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/625189571' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-28T12:49:21.996 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:22.412 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:22 smithi084 ceph-mon[28081]: pgmap v346: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.8 KiB/s wr, 16 op/s 2024-04-28T12:49:22.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:22 smithi161 ceph-mon[62387]: pgmap v346: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.8 KiB/s wr, 16 op/s 2024-04-28T12:49:22.664 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:22 smithi179 ceph-mon[63029]: pgmap v346: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.8 KiB/s wr, 16 op/s 2024-04-28T12:49:24.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.388+0000 7fbd05e00700 1 -- 172.21.15.84:0/3161571241 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd00105640 msgr2=0x7fbd00105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:24.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.388+0000 7fbd05e00700 1 --2- 172.21.15.84:0/3161571241 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd00105640 0x7fbd00105ac0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbce8009990 tx=0x7fbce8009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:24.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.388+0000 7fbd05e00700 1 -- 172.21.15.84:0/3161571241 shutdown_connections 2024-04-28T12:49:24.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.388+0000 7fbd05e00700 1 --2- 172.21.15.84:0/3161571241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd0010b640 0x7fbd0010ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:24.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.388+0000 7fbd05e00700 1 --2- 172.21.15.84:0/3161571241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd00106000 0x7fbd00112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:24.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.388+0000 7fbd05e00700 1 --2- 172.21.15.84:0/3161571241 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd00105640 0x7fbd00105ac0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:24.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.388+0000 7fbd05e00700 1 -- 172.21.15.84:0/3161571241 >> 172.21.15.84:0/3161571241 conn(0x7fbd00101130 msgr2=0x7fbd00103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:24.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.388+0000 7fbd05e00700 1 -- 172.21.15.84:0/3161571241 shutdown_connections 2024-04-28T12:49:24.384 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.388+0000 7fbd05e00700 1 -- 172.21.15.84:0/3161571241 wait complete. 2024-04-28T12:49:24.385 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.389+0000 7fbd05e00700 1 Processor -- start 2024-04-28T12:49:24.385 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.390+0000 7fbd05e00700 1 -- start start 2024-04-28T12:49:24.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.390+0000 7fbd05e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd00105640 0x7fbd001a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:24.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.390+0000 7fbd05e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd00106000 0x7fbd001a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:24.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.390+0000 7fbd05e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd0010b640 0x7fbd0019bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:24.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.390+0000 7fbd05e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fbd00115840 con 0x7fbd0010b640 2024-04-28T12:49:24.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.390+0000 7fbd05e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbd001156c0 con 0x7fbd00106000 2024-04-28T12:49:24.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.390+0000 7fbd05e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fbd001159c0 con 0x7fbd00105640 2024-04-28T12:49:24.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.390+0000 7fbcff400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd0010b640 0x7fbd0019bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:24.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.391+0000 7fbcff400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd0010b640 0x7fbd0019bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:60974/0 (socket says 172.21.15.84:60974) 2024-04-28T12:49:24.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.391+0000 7fbcff400700 1 -- 172.21.15.84:0/1331185470 learned_addr learned my addr 172.21.15.84:0/1331185470 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:24.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.391+0000 7fbcf7e00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd00106000 0x7fbd001a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:24.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.391+0000 7fbcfea00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd00105640 0x7fbd001a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:24.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.391+0000 7fbcfea00700 1 -- 172.21.15.84:0/1331185470 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd00106000 msgr2=0x7fbd001a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:24.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.391+0000 7fbcfea00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd00106000 0x7fbd001a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:24.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.391+0000 7fbcfea00700 1 -- 172.21.15.84:0/1331185470 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd0010b640 msgr2=0x7fbd0019bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:24.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.391+0000 7fbcfea00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd0010b640 0x7fbd0019bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:24.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.391+0000 7fbcfea00700 1 -- 172.21.15.84:0/1331185470 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0019c2a0 con 0x7fbd00105640 2024-04-28T12:49:24.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.391+0000 7fbcff400700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd0010b640 0x7fbd0019bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:49:24.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.391+0000 7fbcfea00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd00105640 0x7fbd001a1930 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000d950 tx=0x7fbcf000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:24.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.392+0000 7fbd04a00700 1 -- 172.21.15.84:0/1331185470 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0017070 con 0x7fbd00105640 2024-04-28T12:49:24.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.392+0000 7fbd05e00700 1 -- 172.21.15.84:0/1331185470 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd0019c590 con 0x7fbd00105640 2024-04-28T12:49:24.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.392+0000 7fbd04a00700 1 -- 172.21.15.84:0/1331185470 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcf0009de0 con 0x7fbd00105640 2024-04-28T12:49:24.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.392+0000 7fbd04a00700 1 -- 172.21.15.84:0/1331185470 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0010d90 con 0x7fbd00105640 2024-04-28T12:49:24.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.392+0000 7fbd05e00700 1 -- 172.21.15.84:0/1331185470 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd001a8800 con 0x7fbd00105640 2024-04-28T12:49:24.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.393+0000 7fbd05e00700 1 -- 172.21.15.84:0/1331185470 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcdc005330 con 0x7fbd00105640 2024-04-28T12:49:24.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.395+0000 7fbd04a00700 1 -- 172.21.15.84:0/1331185470 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbcf001a070 con 0x7fbd00105640 2024-04-28T12:49:24.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.396+0000 7fbd04a00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbce40a3d50 0x7fbce40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:24.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.397+0000 7fbd04a00700 1 -- 172.21.15.84:0/1331185470 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fbcf00cfff0 con 0x7fbd00105640 2024-04-28T12:49:24.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.397+0000 7fbcf7e00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbce40a3d50 0x7fbce40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:24.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.397+0000 7fbcf7e00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbce40a3d50 0x7fbce40a6210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbce80098e0 tx=0x7fbce8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:24.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.401+0000 7fbd04a00700 1 -- 172.21.15.84:0/1331185470 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbcf009c5f0 con 0x7fbd00105640 2024-04-28T12:49:24.625 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.630+0000 7fbd05e00700 1 -- 172.21.15.84:0/1331185470 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbcdc0058f0 con 0x7fbd00105640 2024-04-28T12:49:24.627 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.632+0000 7fbd04a00700 1 -- 172.21.15.84:0/1331185470 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24117 (secure 0 0 0) 0x7fbcf009bef0 con 0x7fbd00105640 2024-04-28T12:49:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:49:24.628 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":90,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:48:40.059255+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:49:24.631 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.635+0000 7fbd05e00700 1 -- 172.21.15.84:0/1331185470 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbce40a3d50 msgr2=0x7fbce40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:24.631 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.635+0000 7fbd05e00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbce40a3d50 0x7fbce40a6210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbce80098e0 tx=0x7fbce8005960 comp rx=0 tx=0).stop 2024-04-28T12:49:24.631 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.635+0000 7fbd05e00700 1 -- 172.21.15.84:0/1331185470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd00105640 msgr2=0x7fbd001a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:24.631 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.635+0000 7fbd05e00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd00105640 0x7fbd001a1930 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000d950 tx=0x7fbcf000dc60 comp rx=0 tx=0).stop 2024-04-28T12:49:24.632 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.635+0000 7fbd05e00700 1 -- 172.21.15.84:0/1331185470 shutdown_connections 2024-04-28T12:49:24.632 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.635+0000 7fbd05e00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd0010b640 0x7fbd0019bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:24.632 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.635+0000 7fbd05e00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd00105640 0x7fbd001a1930 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:24.632 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.635+0000 7fbd05e00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fbce40a3d50 0x7fbce40a6210 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:24.632 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.635+0000 7fbd05e00700 1 --2- 172.21.15.84:0/1331185470 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd00106000 0x7fbd001a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:24.632 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.635+0000 7fbd05e00700 1 -- 172.21.15.84:0/1331185470 >> 172.21.15.84:0/1331185470 conn(0x7fbd00101130 msgr2=0x7fbd00101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:24.632 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.636+0000 7fbd05e00700 1 -- 172.21.15.84:0/1331185470 shutdown_connections 2024-04-28T12:49:24.632 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:24.636+0000 7fbd05e00700 1 -- 172.21.15.84:0/1331185470 wait complete. 2024-04-28T12:49:24.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:24 smithi084 ceph-mon[28081]: pgmap v347: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-04-28T12:49:24.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:24 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1331185470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:24.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:24 smithi161 ceph-mon[62387]: pgmap v347: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-04-28T12:49:24.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:24 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1331185470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:25.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:24 smithi179 ceph-mon[63029]: pgmap v347: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-04-28T12:49:25.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:24 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1331185470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:25.501 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-28T12:49:25.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-28T12:49:25.949 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:25.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:25 smithi161 ceph-mon[62387]: pgmap v348: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.8 KiB/s wr, 9 op/s 2024-04-28T12:49:26.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:25 smithi084 ceph-mon[28081]: pgmap v348: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.8 KiB/s wr, 9 op/s 2024-04-28T12:49:26.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:25 smithi179 ceph-mon[63029]: pgmap v348: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.8 KiB/s wr, 9 op/s 2024-04-28T12:49:28.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.318+0000 7ffac5a00700 1 -- 172.21.15.84:0/3598612777 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffac0109960 msgr2=0x7ffac0109de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:28.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.318+0000 7ffac5a00700 1 --2- 172.21.15.84:0/3598612777 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffac0109960 0x7ffac0109de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009990 tx=0x7ffaac009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.318+0000 7ffac5a00700 1 -- 172.21.15.84:0/3598612777 shutdown_connections 2024-04-28T12:49:28.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.318+0000 7ffac5a00700 1 --2- 172.21.15.84:0/3598612777 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffac010e830 0x7ffac010ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.318+0000 7ffac5a00700 1 --2- 172.21.15.84:0/3598612777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffac010a320 0x7ffac01167c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.318+0000 7ffac5a00700 1 --2- 172.21.15.84:0/3598612777 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffac0109960 0x7ffac0109de0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.318+0000 7ffac5a00700 1 -- 172.21.15.84:0/3598612777 >> 172.21.15.84:0/3598612777 conn(0x7ffac00791b0 msgr2=0x7ffac0108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:28.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.318+0000 7ffac5a00700 1 -- 172.21.15.84:0/3598612777 shutdown_connections 2024-04-28T12:49:28.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.318+0000 7ffac5a00700 1 -- 172.21.15.84:0/3598612777 wait complete. 2024-04-28T12:49:28.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.319+0000 7ffac5a00700 1 Processor -- start 2024-04-28T12:49:28.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.319+0000 7ffac5a00700 1 -- start start 2024-04-28T12:49:28.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.320+0000 7ffac5a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffac0109960 0x7ffac01a0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:28.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.320+0000 7ffac5a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffac010a320 0x7ffac01a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:28.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.320+0000 7ffac5a00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffac010e830 0x7ffac01a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:28.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.320+0000 7ffac5a00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ffac0119b80 con 0x7ffac0109960 2024-04-28T12:49:28.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.320+0000 7ffac5a00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffac0119a00 con 0x7ffac010e830 2024-04-28T12:49:28.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.320+0000 7ffac5a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ffac0119d00 con 0x7ffac010a320 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.320+0000 7ffabfe00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffac0109960 0x7ffac01a0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.320+0000 7ffabfe00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffac0109960 0x7ffac01a0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:60994/0 (socket says 172.21.15.84:60994) 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.320+0000 7ffabfe00700 1 -- 172.21.15.84:0/1842508682 learned_addr learned my addr 172.21.15.84:0/1842508682 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.320+0000 7ffabf400700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffac010a320 0x7ffac01a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.320+0000 7ffac5000700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffac010e830 0x7ffac01a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.321+0000 7ffabfe00700 1 -- 172.21.15.84:0/1842508682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffac010a320 msgr2=0x7ffac01a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.321+0000 7ffabfe00700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffac010a320 0x7ffac01a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.321+0000 7ffabfe00700 1 -- 172.21.15.84:0/1842508682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffac010e830 msgr2=0x7ffac01a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.321+0000 7ffabfe00700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffac010e830 0x7ffac01a4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.321+0000 7ffabfe00700 1 -- 172.21.15.84:0/1842508682 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac01a5060 con 0x7ffac0109960 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.321+0000 7ffabfe00700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffac0109960 0x7ffac01a0050 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffab000e9a0 tx=0x7ffab000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.321+0000 7ffabcc00700 1 -- 172.21.15.84:0/1842508682 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab000ca10 con 0x7ffac0109960 2024-04-28T12:49:28.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.321+0000 7ffabcc00700 1 -- 172.21.15.84:0/1842508682 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab000cbb0 con 0x7ffac0109960 2024-04-28T12:49:28.319 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.321+0000 7ffac5a00700 1 -- 172.21.15.84:0/1842508682 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac01a5350 con 0x7ffac0109960 2024-04-28T12:49:28.319 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.321+0000 7ffabcc00700 1 -- 172.21.15.84:0/1842508682 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0018680 con 0x7ffac0109960 2024-04-28T12:49:28.319 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.322+0000 7ffac5a00700 1 -- 172.21.15.84:0/1842508682 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac01acbd0 con 0x7ffac0109960 2024-04-28T12:49:28.319 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.323+0000 7ffac5a00700 1 -- 172.21.15.84:0/1842508682 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffac006a110 con 0x7ffac0109960 2024-04-28T12:49:28.320 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.325+0000 7ffabcc00700 1 -- 172.21.15.84:0/1842508682 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ffab0010430 con 0x7ffac0109960 2024-04-28T12:49:28.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.326+0000 7ffabcc00700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffaa40a3cd0 0x7ffaa40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:28.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.326+0000 7ffabcc00700 1 -- 172.21.15.84:0/1842508682 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ffab00d0910 con 0x7ffac0109960 2024-04-28T12:49:28.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.326+0000 7ffabf400700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffaa40a3cd0 0x7ffaa40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:28.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.327+0000 7ffabf400700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffaa40a3cd0 0x7ffaa40a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffac0109490 tx=0x7ffaac00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:28.326 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.331+0000 7ffabcc00700 1 -- 172.21.15.84:0/1842508682 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffab009cef0 con 0x7ffac0109960 2024-04-28T12:49:28.555 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.559+0000 7ffac5a00700 1 -- 172.21.15.84:0/1842508682 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffac002e490 con 0x7ffac0109960 2024-04-28T12:49:28.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.562+0000 7ffabcc00700 1 -- 172.21.15.84:0/1842508682 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24117 (secure 0 0 0) 0x7ffab009c7f0 con 0x7ffac0109960 2024-04-28T12:49:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:49:28.559 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":90,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:48:40.059255+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:49:28.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.565+0000 7ffac5a00700 1 -- 172.21.15.84:0/1842508682 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffaa40a3cd0 msgr2=0x7ffaa40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:28.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.565+0000 7ffac5a00700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffaa40a3cd0 0x7ffaa40a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffac0109490 tx=0x7ffaac00b260 comp rx=0 tx=0).stop 2024-04-28T12:49:28.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.565+0000 7ffac5a00700 1 -- 172.21.15.84:0/1842508682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffac0109960 msgr2=0x7ffac01a0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:28.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.565+0000 7ffac5a00700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffac0109960 0x7ffac01a0050 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffab000e9a0 tx=0x7ffab000ecb0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.566+0000 7ffac5a00700 1 -- 172.21.15.84:0/1842508682 shutdown_connections 2024-04-28T12:49:28.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.566+0000 7ffac5a00700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffaa40a3cd0 0x7ffaa40a6190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.566+0000 7ffac5a00700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffac010e830 0x7ffac01a4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.566+0000 7ffac5a00700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffac0109960 0x7ffac01a0050 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.566+0000 7ffac5a00700 1 --2- 172.21.15.84:0/1842508682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffac010a320 0x7ffac01a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:28.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.566+0000 7ffac5a00700 1 -- 172.21.15.84:0/1842508682 >> 172.21.15.84:0/1842508682 conn(0x7ffac00791b0 msgr2=0x7ffac00797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:28.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.566+0000 7ffac5a00700 1 -- 172.21.15.84:0/1842508682 shutdown_connections 2024-04-28T12:49:28.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:28.566+0000 7ffac5a00700 1 -- 172.21.15.84:0/1842508682 wait complete. 2024-04-28T12:49:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:28 smithi084 ceph-mon[28081]: pgmap v349: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-28T12:49:28.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:28 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1842508682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:28.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:28 smithi161 ceph-mon[62387]: pgmap v349: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-28T12:49:28.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:28 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1842508682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:29.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:28 smithi179 ceph-mon[63029]: pgmap v349: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-28T12:49:29.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:28 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1842508682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:49:29.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-28T12:49:29.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-28T12:49:29.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-28T12:49:29.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-28T12:49:29.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-28T12:49:29.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-28T12:49:29.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-28T12:49:29.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-28T12:49:29.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-28T12:49:29.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-28T12:49:29.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-28T12:49:29.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-28T12:49:29.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-28T12:49:29.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-28T12:49:29.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-04-28T12:49:29.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-28T12:49:29.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-04-28T12:49:29.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-28T12:49:29.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-04-28T12:49:29.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 10]}]: dispatch 2024-04-28T12:49:29.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-04-28T12:49:29.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-28T12:49:29.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-28T12:49:29.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-28T12:49:29.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-04-28T12:49:29.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-28T12:49:29.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-04-28T12:49:29.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-28T12:49:29.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-04-28T12:49:29.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 10]}]: dispatch 2024-04-28T12:49:29.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-04-28T12:49:29.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-28T12:49:30.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-28T12:49:30.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-28T12:49:30.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-04-28T12:49:30.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-28T12:49:30.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-04-28T12:49:30.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-28T12:49:30.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-04-28T12:49:30.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 10]}]: dispatch 2024-04-28T12:49:30.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-04-28T12:49:30.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-28T12:49:30.507 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:30.517 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:30.527 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:30.544 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:30.563 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:30.615 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:30.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:30 smithi084 ceph-mon[28081]: pgmap v350: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-28T12:49:30.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-28T12:49:30.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-28T12:49:30.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-04-28T12:49:30.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-28T12:49:30.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-04-28T12:49:30.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-28T12:49:30.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-04-28T12:49:30.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 10]}]': finished 2024-04-28T12:49:30.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-04-28T12:49:30.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-28T12:49:30.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:30 smithi084 ceph-mon[28081]: osdmap e91: 12 total, 12 up, 12 in 2024-04-28T12:49:30.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:30 smithi161 ceph-mon[62387]: pgmap v350: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-28T12:49:30.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-28T12:49:30.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-28T12:49:30.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-04-28T12:49:30.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-28T12:49:30.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-04-28T12:49:30.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-28T12:49:30.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-04-28T12:49:30.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 10]}]': finished 2024-04-28T12:49:30.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-04-28T12:49:30.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-28T12:49:30.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:30 smithi161 ceph-mon[62387]: osdmap e91: 12 total, 12 up, 12 in 2024-04-28T12:49:31.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:30 smithi179 ceph-mon[63029]: pgmap v350: 129 pgs: 129 active+clean; 629 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-28T12:49:31.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-28T12:49:31.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-28T12:49:31.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-04-28T12:49:31.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-28T12:49:31.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-04-28T12:49:31.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-28T12:49:31.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-04-28T12:49:31.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 10]}]': finished 2024-04-28T12:49:31.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-04-28T12:49:31.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-28T12:49:31.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:30 smithi179 ceph-mon[63029]: osdmap e91: 12 total, 12 up, 12 in 2024-04-28T12:49:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:31 smithi084 ceph-mon[28081]: osdmap e92: 12 total, 12 up, 12 in 2024-04-28T12:49:31.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:31 smithi084 ceph-mon[28081]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 639 B/s wr, 7 op/s 2024-04-28T12:49:31.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:31 smithi161 ceph-mon[62387]: osdmap e92: 12 total, 12 up, 12 in 2024-04-28T12:49:31.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:31 smithi161 ceph-mon[62387]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 639 B/s wr, 7 op/s 2024-04-28T12:49:32.132 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:32.132 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:32.134 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:32.135 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:31 smithi179 ceph-mon[63029]: osdmap e92: 12 total, 12 up, 12 in 2024-04-28T12:49:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:31 smithi179 ceph-mon[63029]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 639 B/s wr, 7 op/s 2024-04-28T12:49:32.243 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:32.827 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:49:33.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:33 smithi084 ceph-mon[28081]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 639 B/s wr, 4 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:33.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:33 smithi161 ceph-mon[62387]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 639 B/s wr, 4 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:34.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:33 smithi179 ceph-mon[63029]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 639 B/s wr, 4 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:35.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:49:35.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:49:35.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:49:35.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:35 smithi084 ceph-mon[28081]: pgmap v355: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 639 B/s wr, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:35.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:35 smithi161 ceph-mon[62387]: pgmap v355: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 639 B/s wr, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:36.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:35 smithi179 ceph-mon[63029]: pgmap v355: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 639 B/s wr, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:37.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:49:37.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:49:37.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:49:37.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:49:37.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:49:37.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:49:37.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:49:37.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:49:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:49:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:49:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:49:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:49:38.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:38 smithi179 ceph-mon[63029]: pgmap v356: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 639 B/s wr, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:38.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:38 smithi084 ceph-mon[28081]: pgmap v356: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 639 B/s wr, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:38.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:38 smithi161 ceph-mon[62387]: pgmap v356: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 639 B/s wr, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-28T12:49:39.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:39 smithi161 ceph-mon[62387]: pgmap v357: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-28T12:49:40.049 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:39 smithi084 ceph-mon[28081]: pgmap v357: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-28T12:49:40.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:39 smithi179 ceph-mon[63029]: pgmap v357: 129 pgs: 129 active+clean; 633 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-28T12:49:41.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.023+0000 7fb057800700 1 -- 172.21.15.84:0/4184417788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb050103470 msgr2=0x7fb0501038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:41.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.023+0000 7fb057800700 1 --2- 172.21.15.84:0/4184417788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb050103470 0x7fb0501038f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb03c009990 tx=0x7fb03c009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:41.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.023+0000 7fb057800700 1 -- 172.21.15.84:0/4184417788 shutdown_connections 2024-04-28T12:49:41.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.023+0000 7fb057800700 1 --2- 172.21.15.84:0/4184417788 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0501094c0 0x7fb0501098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:41.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.023+0000 7fb057800700 1 --2- 172.21.15.84:0/4184417788 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb050103e30 0x7fb050110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:41.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.023+0000 7fb057800700 1 --2- 172.21.15.84:0/4184417788 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb050103470 0x7fb0501038f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:41.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.023+0000 7fb057800700 1 -- 172.21.15.84:0/4184417788 >> 172.21.15.84:0/4184417788 conn(0x7fb05007a580 msgr2=0x7fb05007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:41.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.023+0000 7fb057800700 1 -- 172.21.15.84:0/4184417788 shutdown_connections 2024-04-28T12:49:41.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.023+0000 7fb057800700 1 -- 172.21.15.84:0/4184417788 wait complete. 2024-04-28T12:49:41.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.024+0000 7fb057800700 1 Processor -- start 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.025+0000 7fb057800700 1 -- start start 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.025+0000 7fb057800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb050103470 0x7fb05019bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.025+0000 7fb057800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb050103e30 0x7fb05019c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.025+0000 7fb057800700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0501094c0 0x7fb0501a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.025+0000 7fb057800700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb050113610 con 0x7fb050103e30 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.025+0000 7fb057800700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb050113490 con 0x7fb0501094c0 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.025+0000 7fb057800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb050113790 con 0x7fb050103470 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.025+0000 7fb054e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb050103470 0x7fb05019bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.025+0000 7fb047e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb050103e30 0x7fb05019c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.025+0000 7fb054e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb050103470 0x7fb05019bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:35880/0 (socket says 172.21.15.84:35880) 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.025+0000 7fb054e00700 1 -- 172.21.15.84:0/1780182511 learned_addr learned my addr 172.21.15.84:0/1780182511 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.026+0000 7fb055800700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0501094c0 0x7fb0501a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:41.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.026+0000 7fb054e00700 1 -- 172.21.15.84:0/1780182511 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0501094c0 msgr2=0x7fb0501a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:41.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.026+0000 7fb054e00700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0501094c0 0x7fb0501a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:41.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.026+0000 7fb054e00700 1 -- 172.21.15.84:0/1780182511 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb050103e30 msgr2=0x7fb05019c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:41.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.026+0000 7fb054e00700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb050103e30 0x7fb05019c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:41.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.026+0000 7fb054e00700 1 -- 172.21.15.84:0/1780182511 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0501a0d70 con 0x7fb050103470 2024-04-28T12:49:41.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.026+0000 7fb055800700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0501094c0 0x7fb0501a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:49:41.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.026+0000 7fb054e00700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb050103470 0x7fb05019bd60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb04000e9a0 tx=0x7fb04000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:41.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.026+0000 7fb04e000700 1 -- 172.21.15.84:0/1780182511 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04000cbd0 con 0x7fb050103470 2024-04-28T12:49:41.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.027+0000 7fb057800700 1 -- 172.21.15.84:0/1780182511 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0501a1060 con 0x7fb050103470 2024-04-28T12:49:41.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.027+0000 7fb04e000700 1 -- 172.21.15.84:0/1780182511 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb04000cd70 con 0x7fb050103470 2024-04-28T12:49:41.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.027+0000 7fb04e000700 1 -- 172.21.15.84:0/1780182511 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb040018680 con 0x7fb050103470 2024-04-28T12:49:41.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.027+0000 7fb057800700 1 -- 172.21.15.84:0/1780182511 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0501a88e0 con 0x7fb050103470 2024-04-28T12:49:41.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.028+0000 7fb046a00700 1 -- 172.21.15.84:0/1780182511 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb050063bb0 con 0x7fb050103470 2024-04-28T12:49:41.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.030+0000 7fb04e000700 1 -- 172.21.15.84:0/1780182511 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb040010430 con 0x7fb050103470 2024-04-28T12:49:41.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.031+0000 7fb04e000700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0340a3cd0 0x7fb0340a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:41.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.031+0000 7fb04e000700 1 -- 172.21.15.84:0/1780182511 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fb0400d0d50 con 0x7fb050103470 2024-04-28T12:49:41.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.031+0000 7fb047e00700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0340a3cd0 0x7fb0340a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:41.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.032+0000 7fb04e000700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785] conn(0x7fb0340aee30 0x7fb0340b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:41.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.032+0000 7fb04e000700 1 -- 172.21.15.84:0/1780182511 --> [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb04001bec0 con 0x7fb0340aee30 2024-04-28T12:49:41.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.032+0000 7fb04e000700 1 -- 172.21.15.84:0/1780182511 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fb0400d1100 con 0x7fb050103470 2024-04-28T12:49:41.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.032+0000 7fb055800700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785] conn(0x7fb0340aee30 0x7fb0340b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:41.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.032+0000 7fb047e00700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0340a3cd0 0x7fb0340a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb05007ae80 tx=0x7fb03c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:41.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.033+0000 7fb055800700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785] conn(0x7fb0340aee30 0x7fb0340b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:41.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.034+0000 7fb04e000700 1 -- 172.21.15.84:0/1780182511 <== osd.6 v2:172.21.15.84:6818/440861785 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb04001bec0 con 0x7fb0340aee30 2024-04-28T12:49:41.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.052+0000 7fb046a00700 1 -- 172.21.15.84:0/1780182511 --> [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb050063fb0 con 0x7fb0340aee30 2024-04-28T12:49:41.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.052+0000 7fb04e000700 1 -- 172.21.15.84:0/1780182511 <== osd.6 v2:172.21.15.84:6818/440861785 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb050063fb0 con 0x7fb0340aee30 2024-04-28T12:49:41.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.053+0000 7fb046a00700 1 -- 172.21.15.84:0/1780182511 >> [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785] conn(0x7fb0340aee30 msgr2=0x7fb0340b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:41.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.053+0000 7fb046a00700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785] conn(0x7fb0340aee30 0x7fb0340b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:41.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.053+0000 7fb046a00700 1 -- 172.21.15.84:0/1780182511 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0340a3cd0 msgr2=0x7fb0340a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:41.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.053+0000 7fb046a00700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0340a3cd0 0x7fb0340a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb05007ae80 tx=0x7fb03c00b260 comp rx=0 tx=0).stop 2024-04-28T12:49:41.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.053+0000 7fb046a00700 1 -- 172.21.15.84:0/1780182511 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb050103470 msgr2=0x7fb05019bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:41.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.053+0000 7fb046a00700 1 --2- 172.21.15.84:0/1780182511 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb050103470 0x7fb05019bd60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb04000e9a0 tx=0x7fb04000ecb0 comp rx=0 tx=0).stop 2024-04-28T12:49:41.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.053+0000 7fb055800700 1 -- 172.21.15.84:0/1780182511 reap_dead start 2024-04-28T12:49:41.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.053+0000 7fb046a00700 1 -- 172.21.15.84:0/1780182511 shutdown_connections 2024-04-28T12:49:41.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.053+0000 7fb046a00700 1 -- 172.21.15.84:0/1780182511 >> 172.21.15.84:0/1780182511 conn(0x7fb05007a580 msgr2=0x7fb05010e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:41.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.054+0000 7fb046a00700 1 -- 172.21.15.84:0/1780182511 shutdown_connections 2024-04-28T12:49:41.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:41.054+0000 7fb046a00700 1 -- 172.21.15.84:0/1780182511 wait complete. 2024-04-28T12:49:41.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:41 smithi084 ceph-mon[28081]: pgmap v358: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 187 B/s wr, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-28T12:49:41.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:41 smithi161 ceph-mon[62387]: pgmap v358: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 187 B/s wr, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-28T12:49:42.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:41 smithi179 ceph-mon[63029]: pgmap v358: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 187 B/s wr, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-28T12:49:43.882 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:43 smithi084 ceph-mon[28081]: pgmap v359: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-28T12:49:43.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:43 smithi161 ceph-mon[62387]: pgmap v359: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-28T12:49:44.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:43 smithi179 ceph-mon[63029]: pgmap v359: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-28T12:49:44.737 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.740+0000 7f3875000700 1 -- 172.21.15.84:0/1692981613 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f387010a410 msgr2=0x7f3870116920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:44.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.740+0000 7f3875000700 1 --2- 172.21.15.84:0/1692981613 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f387010a410 0x7f3870116920 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f386800b0a0 tx=0x7f386800b3b0 comp rx=0 tx=0).stop 2024-04-28T12:49:44.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.740+0000 7f3875000700 1 -- 172.21.15.84:0/1692981613 shutdown_connections 2024-04-28T12:49:44.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.740+0000 7f3875000700 1 --2- 172.21.15.84:0/1692981613 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f387010a410 0x7f3870116920 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:44.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.740+0000 7f3875000700 1 --2- 172.21.15.84:0/1692981613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f387010fa50 0x7f387010fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:44.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.740+0000 7f3875000700 1 --2- 172.21.15.84:0/1692981613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3870109a50 0x7f3870109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:44.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.740+0000 7f3875000700 1 -- 172.21.15.84:0/1692981613 >> 172.21.15.84:0/1692981613 conn(0x7f3870076bf0 msgr2=0x7f3870108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:44.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.740+0000 7f3875000700 1 -- 172.21.15.84:0/1692981613 shutdown_connections 2024-04-28T12:49:44.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.740+0000 7f3875000700 1 -- 172.21.15.84:0/1692981613 wait complete. 2024-04-28T12:49:44.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.741+0000 7f3875000700 1 Processor -- start 2024-04-28T12:49:44.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.742+0000 7f3875000700 1 -- start start 2024-04-28T12:49:44.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.742+0000 7f3875000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3870109a50 0x7f38701a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:44.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.742+0000 7f3875000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f387010a410 0x7f38701a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:44.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.742+0000 7f3875000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f387010fa50 0x7f38701a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:44.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.742+0000 7f3875000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3870119ce0 con 0x7f3870109a50 2024-04-28T12:49:44.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.742+0000 7f3875000700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3870119b60 con 0x7f387010fa50 2024-04-28T12:49:44.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.742+0000 7f3875000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3870119e60 con 0x7f387010a410 2024-04-28T12:49:44.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.742+0000 7f386de00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3870109a50 0x7f38701a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:44.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.742+0000 7f386de00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3870109a50 0x7f38701a00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55318/0 (socket says 172.21.15.84:55318) 2024-04-28T12:49:44.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.742+0000 7f386de00700 1 -- 172.21.15.84:0/1395695601 learned_addr learned my addr 172.21.15.84:0/1395695601 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:44.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.742+0000 7f386e800700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f387010fa50 0x7f38701a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:44.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.743+0000 7f386d400700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f387010a410 0x7f38701a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:44.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.743+0000 7f386de00700 1 -- 172.21.15.84:0/1395695601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f387010a410 msgr2=0x7f38701a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:44.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.743+0000 7f386de00700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f387010a410 0x7f38701a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:44.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.743+0000 7f386de00700 1 -- 172.21.15.84:0/1395695601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f387010fa50 msgr2=0x7f38701a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:44.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.743+0000 7f386de00700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f387010fa50 0x7f38701a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:44.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.743+0000 7f386de00700 1 -- 172.21.15.84:0/1395695601 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38701a5090 con 0x7f3870109a50 2024-04-28T12:49:44.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.743+0000 7f386de00700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3870109a50 0x7f38701a00d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f386000d950 tx=0x7f386000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:44.742 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.743+0000 7f3866a00700 1 -- 172.21.15.84:0/1395695601 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3860016070 con 0x7f3870109a50 2024-04-28T12:49:44.742 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.744+0000 7f3866a00700 1 -- 172.21.15.84:0/1395695601 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f386000fd40 con 0x7f3870109a50 2024-04-28T12:49:44.742 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.744+0000 7f3875000700 1 -- 172.21.15.84:0/1395695601 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38701a5380 con 0x7f3870109a50 2024-04-28T12:49:44.742 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.744+0000 7f3866a00700 1 -- 172.21.15.84:0/1395695601 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386000e3b0 con 0x7f3870109a50 2024-04-28T12:49:44.742 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.744+0000 7f3875000700 1 -- 172.21.15.84:0/1395695601 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38701acc00 con 0x7f3870109a50 2024-04-28T12:49:44.742 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.745+0000 7f3875000700 1 -- 172.21.15.84:0/1395695601 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3870063bb0 con 0x7f3870109a50 2024-04-28T12:49:44.743 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.747+0000 7f3866a00700 1 -- 172.21.15.84:0/1395695601 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3860019070 con 0x7f3870109a50 2024-04-28T12:49:44.744 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.748+0000 7f3866a00700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f38540a3d50 0x7f38540a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:44.745 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.748+0000 7f3866a00700 1 -- 172.21.15.84:0/1395695601 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f38600d0820 con 0x7f3870109a50 2024-04-28T12:49:44.745 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.748+0000 7f386d400700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f38540a3d50 0x7f38540a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:44.745 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.749+0000 7f3866a00700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287] conn(0x7f38540aeeb0 0x7f38540b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:44.745 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.749+0000 7f3866a00700 1 -- 172.21.15.84:0/1395695601 --> [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3860013ec0 con 0x7f38540aeeb0 2024-04-28T12:49:44.745 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.749+0000 7f3866a00700 1 -- 172.21.15.84:0/1395695601 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f38600d0bd0 con 0x7f3870109a50 2024-04-28T12:49:44.746 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.749+0000 7f386e800700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287] conn(0x7f38540aeeb0 0x7f38540b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:44.746 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.749+0000 7f386d400700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f38540a3d50 0x7f38540a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3870078a80 tx=0x7f385c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:44.746 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.750+0000 7f386e800700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287] conn(0x7f38540aeeb0 0x7f38540b1350 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:44.747 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.750+0000 7f3866a00700 1 -- 172.21.15.84:0/1395695601 <== osd.0 v2:172.21.15.84:6802/598031287 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3860013ec0 con 0x7f38540aeeb0 2024-04-28T12:49:44.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.768+0000 7f3875000700 1 -- 172.21.15.84:0/1395695601 --> [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3870063fe0 con 0x7f38540aeeb0 2024-04-28T12:49:44.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.769+0000 7f3866a00700 1 -- 172.21.15.84:0/1395695601 <== osd.0 v2:172.21.15.84:6802/598031287 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3870063fe0 con 0x7f38540aeeb0 2024-04-28T12:49:44.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.769+0000 7f3875000700 1 -- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287] conn(0x7f38540aeeb0 msgr2=0x7f38540b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:44.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.769+0000 7f3875000700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287] conn(0x7f38540aeeb0 0x7f38540b1350 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:44.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.770+0000 7f3875000700 1 -- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f38540a3d50 msgr2=0x7f38540a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:44.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.770+0000 7f3875000700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f38540a3d50 0x7f38540a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3870078a80 tx=0x7f385c00b410 comp rx=0 tx=0).stop 2024-04-28T12:49:44.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.770+0000 7f3875000700 1 -- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3870109a50 msgr2=0x7f38701a00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:44.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.770+0000 7f3875000700 1 --2- 172.21.15.84:0/1395695601 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3870109a50 0x7f38701a00d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f386000d950 tx=0x7f386000dc60 comp rx=0 tx=0).stop 2024-04-28T12:49:44.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.770+0000 7f386e800700 1 -- 172.21.15.84:0/1395695601 reap_dead start 2024-04-28T12:49:44.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.770+0000 7f3875000700 1 -- 172.21.15.84:0/1395695601 shutdown_connections 2024-04-28T12:49:44.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.770+0000 7f3875000700 1 -- 172.21.15.84:0/1395695601 >> 172.21.15.84:0/1395695601 conn(0x7f3870076bf0 msgr2=0x7f3870077110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:44.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.770+0000 7f3875000700 1 -- 172.21.15.84:0/1395695601 shutdown_connections 2024-04-28T12:49:44.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:44.770+0000 7f3875000700 1 -- 172.21.15.84:0/1395695601 wait complete. 2024-04-28T12:49:45.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:45 smithi084 ceph-mon[28081]: pgmap v360: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-28T12:49:45.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:45 smithi161 ceph-mon[62387]: pgmap v360: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-28T12:49:46.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:45 smithi179 ceph-mon[63029]: pgmap v360: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-28T12:49:46.166 INFO:teuthology.orchestra.run.smithi084.stdout:197568495672 2024-04-28T12:49:46.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-28T12:49:46.977 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.829+0000 7fd5c4a00700 1 -- 172.21.15.84:0/496434243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c006ba50 msgr2=0x7fd5c010f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:46.977 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.829+0000 7fd5c4a00700 1 --2- 172.21.15.84:0/496434243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c006ba50 0x7fd5c010f990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009990 tx=0x7fd5ac009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:46.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.830+0000 7fd5c4a00700 1 -- 172.21.15.84:0/496434243 shutdown_connections 2024-04-28T12:49:46.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.830+0000 7fd5c4a00700 1 --2- 172.21.15.84:0/496434243 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5c006b130 0x7fd5c006b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:46.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.830+0000 7fd5c4a00700 1 --2- 172.21.15.84:0/496434243 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c010fed0 0x7fd5c01122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:46.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.830+0000 7fd5c4a00700 1 --2- 172.21.15.84:0/496434243 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c006ba50 0x7fd5c010f990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:46.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.830+0000 7fd5c4a00700 1 -- 172.21.15.84:0/496434243 >> 172.21.15.84:0/496434243 conn(0x7fd5c00792d0 msgr2=0x7fd5c00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:46.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.830+0000 7fd5c4a00700 1 -- 172.21.15.84:0/496434243 shutdown_connections 2024-04-28T12:49:46.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.830+0000 7fd5c4a00700 1 -- 172.21.15.84:0/496434243 wait complete. 2024-04-28T12:49:46.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.831+0000 7fd5c4a00700 1 Processor -- start 2024-04-28T12:49:46.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.831+0000 7fd5c4a00700 1 -- start start 2024-04-28T12:49:46.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.832+0000 7fd5c4a00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5c006b130 0x7fd5c019bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:46.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.832+0000 7fd5c4a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c006ba50 0x7fd5c019c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:46.978 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.832+0000 7fd5c4a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c010fed0 0x7fd5c01a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.832+0000 7fd5c4a00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c0115830 con 0x7fd5c010fed0 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.832+0000 7fd5c4a00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c01156b0 con 0x7fd5c006b130 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.832+0000 7fd5c4a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c01159b0 con 0x7fd5c006ba50 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.832+0000 7fd5bd400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c006ba50 0x7fd5c019c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.832+0000 7fd5bd400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c006ba50 0x7fd5c019c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:35912/0 (socket says 172.21.15.84:35912) 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.832+0000 7fd5bd400700 1 -- 172.21.15.84:0/3381447515 learned_addr learned my addr 172.21.15.84:0/3381447515 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.832+0000 7fd5bde00700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5c006b130 0x7fd5c019bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.833+0000 7fd5bd400700 1 -- 172.21.15.84:0/3381447515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5c006b130 msgr2=0x7fd5c019bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.833+0000 7fd5bd400700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5c006b130 0x7fd5c019bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.833+0000 7fd5bd400700 1 -- 172.21.15.84:0/3381447515 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c010fed0 msgr2=0x7fd5c01a0670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.833+0000 7fd5bd400700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c010fed0 0x7fd5c01a0670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.833+0000 7fd5bd400700 1 -- 172.21.15.84:0/3381447515 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c01a0db0 con 0x7fd5c006ba50 2024-04-28T12:49:46.979 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.833+0000 7fd5bde00700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5c006b130 0x7fd5c019bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:49:46.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.833+0000 7fd5bd400700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c006ba50 0x7fd5c019c2e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac005910 tx=0x7fd5ac00b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:46.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.833+0000 7fd5b6a00700 1 -- 172.21.15.84:0/3381447515 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac027070 con 0x7fd5c006ba50 2024-04-28T12:49:46.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.833+0000 7fd5c4a00700 1 -- 172.21.15.84:0/3381447515 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c01a1040 con 0x7fd5c006ba50 2024-04-28T12:49:46.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.833+0000 7fd5b6a00700 1 -- 172.21.15.84:0/3381447515 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5ac022470 con 0x7fd5c006ba50 2024-04-28T12:49:46.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.833+0000 7fd5b6a00700 1 -- 172.21.15.84:0/3381447515 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac017650 con 0x7fd5c006ba50 2024-04-28T12:49:46.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.833+0000 7fd5c4a00700 1 -- 172.21.15.84:0/3381447515 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c01a8920 con 0x7fd5c006ba50 2024-04-28T12:49:46.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.834+0000 7fd5c4a00700 1 -- 172.21.15.84:0/3381447515 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd5c0063bb0 con 0x7fd5c006ba50 2024-04-28T12:49:46.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.837+0000 7fd5b6a00700 1 -- 172.21.15.84:0/3381447515 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd5ac02a070 con 0x7fd5c006ba50 2024-04-28T12:49:46.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.838+0000 7fd5b6a00700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5a40a3cd0 0x7fd5a40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:46.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.838+0000 7fd5b6a00700 1 -- 172.21.15.84:0/3381447515 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fd5ac0d1e90 con 0x7fd5c006ba50 2024-04-28T12:49:46.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.838+0000 7fd5bde00700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5a40a3cd0 0x7fd5a40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:46.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.838+0000 7fd5b6a00700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087] conn(0x7fd5a40aee30 0x7fd5a40b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:46.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.838+0000 7fd5b6a00700 1 -- 172.21.15.84:0/3381447515 --> [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5ac00b0b0 con 0x7fd5a40aee30 2024-04-28T12:49:46.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.838+0000 7fd5b6a00700 1 -- 172.21.15.84:0/3381447515 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fd5ac0d2240 con 0x7fd5c006ba50 2024-04-28T12:49:46.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.839+0000 7fd5be800700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087] conn(0x7fd5a40aee30 0x7fd5a40b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:46.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.839+0000 7fd5bde00700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5a40a3cd0 0x7fd5a40a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009cb0 tx=0x7fd5b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:46.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.839+0000 7fd5be800700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087] conn(0x7fd5a40aee30 0x7fd5a40b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:46.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.840+0000 7fd5b6a00700 1 -- 172.21.15.84:0/3381447515 <== osd.7 v2:172.21.15.161:6816/1226878087 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd5ac00b0b0 con 0x7fd5a40aee30 2024-04-28T12:49:46.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.859+0000 7fd5c4a00700 1 -- 172.21.15.84:0/3381447515 --> [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd5c0064030 con 0x7fd5a40aee30 2024-04-28T12:49:46.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.859+0000 7fd5b6a00700 1 -- 172.21.15.84:0/3381447515 <== osd.7 v2:172.21.15.161:6816/1226878087 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd5c0064030 con 0x7fd5a40aee30 2024-04-28T12:49:46.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.860+0000 7fd5c4a00700 1 -- 172.21.15.84:0/3381447515 >> [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087] conn(0x7fd5a40aee30 msgr2=0x7fd5a40b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:46.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.860+0000 7fd5c4a00700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087] conn(0x7fd5a40aee30 0x7fd5a40b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:46.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.860+0000 7fd5c4a00700 1 -- 172.21.15.84:0/3381447515 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5a40a3cd0 msgr2=0x7fd5a40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:46.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.860+0000 7fd5c4a00700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5a40a3cd0 0x7fd5a40a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009cb0 tx=0x7fd5b0009260 comp rx=0 tx=0).stop 2024-04-28T12:49:46.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.860+0000 7fd5c4a00700 1 -- 172.21.15.84:0/3381447515 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c006ba50 msgr2=0x7fd5c019c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:46.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.860+0000 7fd5c4a00700 1 --2- 172.21.15.84:0/3381447515 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c006ba50 0x7fd5c019c2e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac005910 tx=0x7fd5ac00b670 comp rx=0 tx=0).stop 2024-04-28T12:49:46.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.860+0000 7fd5be800700 1 -- 172.21.15.84:0/3381447515 reap_dead start 2024-04-28T12:49:46.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.860+0000 7fd5c4a00700 1 -- 172.21.15.84:0/3381447515 shutdown_connections 2024-04-28T12:49:46.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.860+0000 7fd5c4a00700 1 -- 172.21.15.84:0/3381447515 >> 172.21.15.84:0/3381447515 conn(0x7fd5c00792d0 msgr2=0x7fd5c0111400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:46.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.861+0000 7fd5c4a00700 1 -- 172.21.15.84:0/3381447515 shutdown_connections 2024-04-28T12:49:46.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:46.861+0000 7fd5c4a00700 1 -- 172.21.15.84:0/3381447515 wait complete. 2024-04-28T12:49:47.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.827+0000 7f5141000700 1 -- 172.21.15.84:0/3487285137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513c06b9c0 msgr2=0x7f513c06bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:47.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.827+0000 7f5141000700 1 --2- 172.21.15.84:0/3487285137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513c06b9c0 0x7f513c06bda0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f512c009990 tx=0x7f512c009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.827+0000 7f5141000700 1 -- 172.21.15.84:0/3487285137 shutdown_connections 2024-04-28T12:49:47.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.827+0000 7f5141000700 1 --2- 172.21.15.84:0/3487285137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513c06b9c0 0x7f513c06bda0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.827+0000 7f5141000700 1 --2- 172.21.15.84:0/3487285137 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f513c10fec0 0x7f513c1122b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.827+0000 7f5141000700 1 --2- 172.21.15.84:0/3487285137 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f513c06c2e0 0x7f513c10f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.827+0000 7f5141000700 1 -- 172.21.15.84:0/3487285137 >> 172.21.15.84:0/3487285137 conn(0x7f513c079290 msgr2=0x7f513c0796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:47.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.827+0000 7f5141000700 1 -- 172.21.15.84:0/3487285137 shutdown_connections 2024-04-28T12:49:47.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.827+0000 7f5141000700 1 -- 172.21.15.84:0/3487285137 wait complete. 2024-04-28T12:49:47.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.828+0000 7f5141000700 1 Processor -- start 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.828+0000 7f5141000700 1 -- start start 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f5141000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513c06b9c0 0x7f513c19bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f5141000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f513c06c2e0 0x7f513c19c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f5141000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f513c10fec0 0x7f513c1a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f5141000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f513c115820 con 0x7f513c10fec0 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f5141000700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f513c1156a0 con 0x7f513c06b9c0 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f5141000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f513c1159a0 con 0x7f513c06c2e0 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f513a800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f513c10fec0 0x7f513c1a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f513a800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f513c10fec0 0x7f513c1a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55342/0 (socket says 172.21.15.84:55342) 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f513a800700 1 -- 172.21.15.84:0/2613867800 learned_addr learned my addr 172.21.15.84:0/2613867800 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f5139400700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f513c06c2e0 0x7f513c19c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f5139e00700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513c06b9c0 0x7f513c19bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f513a800700 1 -- 172.21.15.84:0/2613867800 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f513c06c2e0 msgr2=0x7f513c19c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:47.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f513a800700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f513c06c2e0 0x7f513c19c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f513a800700 1 -- 172.21.15.84:0/2613867800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513c06b9c0 msgr2=0x7f513c19bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f513a800700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f513c06b9c0 0x7f513c19bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f513a800700 1 -- 172.21.15.84:0/2613867800 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513c1a0d80 con 0x7f513c10fec0 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.829+0000 7f5139400700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f513c06c2e0 0x7f513c19c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.830+0000 7f513a800700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f513c10fec0 0x7f513c1a0640 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f513400c110 tx=0x7f513400c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.830+0000 7f5132a00700 1 -- 172.21.15.84:0/2613867800 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51340092e0 con 0x7f513c10fec0 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.830+0000 7f5141000700 1 -- 172.21.15.84:0/2613867800 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f513c1a1010 con 0x7f513c10fec0 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.830+0000 7f5132a00700 1 -- 172.21.15.84:0/2613867800 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5134003a00 con 0x7f513c10fec0 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.830+0000 7f5132a00700 1 -- 172.21.15.84:0/2613867800 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5134004650 con 0x7f513c10fec0 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.830+0000 7f5141000700 1 -- 172.21.15.84:0/2613867800 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f513c1a88f0 con 0x7f513c10fec0 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.831+0000 7f5141000700 1 -- 172.21.15.84:0/2613867800 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f513c063bb0 con 0x7f513c10fec0 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.834+0000 7f5132a00700 1 -- 172.21.15.84:0/2613867800 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f513401a020 con 0x7f513c10fec0 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.835+0000 7f5132a00700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f51200a3d50 0x7f51200a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:47.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.835+0000 7f5132a00700 1 -- 172.21.15.84:0/2613867800 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f51340d0b80 con 0x7f513c10fec0 2024-04-28T12:49:47.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.835+0000 7f5139e00700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f51200a3d50 0x7f51200a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:47.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.835+0000 7f5132a00700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178] conn(0x7f51200aeeb0 0x7f51200b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:47.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.835+0000 7f5132a00700 1 -- 172.21.15.84:0/2613867800 --> [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5134009c20 con 0x7f51200aeeb0 2024-04-28T12:49:47.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.835+0000 7f5132a00700 1 -- 172.21.15.84:0/2613867800 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f51340d0f30 con 0x7f513c10fec0 2024-04-28T12:49:47.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.836+0000 7f5139400700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178] conn(0x7f51200aeeb0 0x7f51200b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:47.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.836+0000 7f5139e00700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f51200a3d50 0x7f51200a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f512c00b2e0 tx=0x7f512c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:47.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.836+0000 7f5139400700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178] conn(0x7f51200aeeb0 0x7f51200b1350 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:47.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.837+0000 7f5132a00700 1 -- 172.21.15.84:0/2613867800 <== osd.10 v2:172.21.15.161:6824/3240597178 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5134009c20 con 0x7f51200aeeb0 2024-04-28T12:49:47.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.858+0000 7f5141000700 1 -- 172.21.15.84:0/2613867800 --> [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f513c064030 con 0x7f51200aeeb0 2024-04-28T12:49:47.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.858+0000 7f5132a00700 1 -- 172.21.15.84:0/2613867800 <== osd.10 v2:172.21.15.161:6824/3240597178 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f513c064030 con 0x7f51200aeeb0 2024-04-28T12:49:47.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.859+0000 7f5141000700 1 -- 172.21.15.84:0/2613867800 >> [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178] conn(0x7f51200aeeb0 msgr2=0x7f51200b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:47.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.859+0000 7f5141000700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178] conn(0x7f51200aeeb0 0x7f51200b1350 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.859+0000 7f5141000700 1 -- 172.21.15.84:0/2613867800 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f51200a3d50 msgr2=0x7f51200a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:47.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.859+0000 7f5141000700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f51200a3d50 0x7f51200a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f512c00b2e0 tx=0x7f512c005fd0 comp rx=0 tx=0).stop 2024-04-28T12:49:47.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.859+0000 7f5141000700 1 -- 172.21.15.84:0/2613867800 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f513c10fec0 msgr2=0x7f513c1a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:47.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.859+0000 7f5141000700 1 --2- 172.21.15.84:0/2613867800 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f513c10fec0 0x7f513c1a0640 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f513400c110 tx=0x7f513400c420 comp rx=0 tx=0).stop 2024-04-28T12:49:47.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.859+0000 7f513a800700 1 -- 172.21.15.84:0/2613867800 reap_dead start 2024-04-28T12:49:47.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.859+0000 7f5141000700 1 -- 172.21.15.84:0/2613867800 shutdown_connections 2024-04-28T12:49:47.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.859+0000 7f5141000700 1 -- 172.21.15.84:0/2613867800 >> 172.21.15.84:0/2613867800 conn(0x7f513c079290 msgr2=0x7f513c111470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:47.901 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.859+0000 7f5141000700 1 -- 172.21.15.84:0/2613867800 shutdown_connections 2024-04-28T12:49:47.901 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:47.859+0000 7f5141000700 1 -- 172.21.15.84:0/2613867800 wait complete. 2024-04-28T12:49:47.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:47 smithi084 ceph-mon[28081]: pgmap v361: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-28T12:49:47.987 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:47 smithi179 ceph-mon[63029]: pgmap v361: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-28T12:49:47.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:47 smithi161 ceph-mon[62387]: pgmap v361: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-28T12:49:49.591 INFO:teuthology.orchestra.run.smithi084.stdout:34359738457 2024-04-28T12:49:49.592 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-28T12:49:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:49 smithi084 ceph-mon[28081]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-28T12:49:49.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:49 smithi161 ceph-mon[62387]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-28T12:49:50.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:49 smithi179 ceph-mon[63029]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-28T12:49:50.198 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.201+0000 7f4123c00700 1 -- 172.21.15.84:0/2702427494 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f411c1094c0 msgr2=0x7f411c1098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:50.199 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.201+0000 7f4123c00700 1 --2- 172.21.15.84:0/2702427494 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f411c1094c0 0x7f411c1098a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f410c009990 tx=0x7f410c009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.199 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.201+0000 7f4123c00700 1 -- 172.21.15.84:0/2702427494 shutdown_connections 2024-04-28T12:49:50.199 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.201+0000 7f4123c00700 1 --2- 172.21.15.84:0/2702427494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f411c103e30 0x7f411c110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.199 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.201+0000 7f4123c00700 1 --2- 172.21.15.84:0/2702427494 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f411c1094c0 0x7f411c1098a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.199 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.201+0000 7f4123c00700 1 --2- 172.21.15.84:0/2702427494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f411c103470 0x7f411c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.199 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.201+0000 7f4123c00700 1 -- 172.21.15.84:0/2702427494 >> 172.21.15.84:0/2702427494 conn(0x7f411c07a580 msgr2=0x7f411c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:50.199 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.202+0000 7f4123c00700 1 -- 172.21.15.84:0/2702427494 shutdown_connections 2024-04-28T12:49:50.199 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.202+0000 7f4123c00700 1 -- 172.21.15.84:0/2702427494 wait complete. 2024-04-28T12:49:50.200 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.203+0000 7f4123c00700 1 Processor -- start 2024-04-28T12:49:50.200 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.203+0000 7f4123c00700 1 -- start start 2024-04-28T12:49:50.201 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.204+0000 7f4123c00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f411c103470 0x7f411c1a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:50.201 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.204+0000 7f4123c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f411c103e30 0x7f411c1a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:50.201 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.204+0000 7f4123c00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f411c1094c0 0x7f411c19bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:50.201 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.204+0000 7f4123c00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f411c1135f0 con 0x7f411c103470 2024-04-28T12:49:50.201 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.204+0000 7f4123c00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f411c113470 con 0x7f411c1094c0 2024-04-28T12:49:50.201 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.204+0000 7f4123c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f411c113770 con 0x7f411c103e30 2024-04-28T12:49:50.201 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.204+0000 7f4121200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f411c103470 0x7f411c1a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:50.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.204+0000 7f4121200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f411c103470 0x7f411c1a1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:48986/0 (socket says 172.21.15.84:48986) 2024-04-28T12:49:50.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.204+0000 7f4121200700 1 -- 172.21.15.84:0/2245774528 learned_addr learned my addr 172.21.15.84:0/2245774528 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:50.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.204+0000 7f4113e00700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f411c103e30 0x7f411c1a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:50.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.204+0000 7f4121c00700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f411c1094c0 0x7f411c19bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.205+0000 7f4113e00700 1 -- 172.21.15.84:0/2245774528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f411c1094c0 msgr2=0x7f411c19bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.205+0000 7f4113e00700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f411c1094c0 0x7f411c19bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.205+0000 7f4113e00700 1 -- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f411c103470 msgr2=0x7f411c1a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.205+0000 7f4113e00700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f411c103470 0x7f411c1a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.205+0000 7f4113e00700 1 -- 172.21.15.84:0/2245774528 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411c19c400 con 0x7f411c103e30 2024-04-28T12:49:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.205+0000 7f4121200700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f411c103470 0x7f411c1a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.205+0000 7f4113e00700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f411c103e30 0x7f411c1a1f90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f410800e9f0 tx=0x7f410800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.205+0000 7f411a000700 1 -- 172.21.15.84:0/2245774528 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4108004220 con 0x7f411c103e30 2024-04-28T12:49:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.206+0000 7f411a000700 1 -- 172.21.15.84:0/2245774528 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41080043c0 con 0x7f411c103e30 2024-04-28T12:49:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.206+0000 7f4123c00700 1 -- 172.21.15.84:0/2245774528 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f411c19c6f0 con 0x7f411c103e30 2024-04-28T12:49:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.206+0000 7f411a000700 1 -- 172.21.15.84:0/2245774528 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4108005020 con 0x7f411c103e30 2024-04-28T12:49:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.206+0000 7f4123c00700 1 -- 172.21.15.84:0/2245774528 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f411c1a89d0 con 0x7f411c103e30 2024-04-28T12:49:50.204 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.207+0000 7f4123c00700 1 -- 172.21.15.84:0/2245774528 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f411c063b30 con 0x7f411c103e30 2024-04-28T12:49:50.206 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.209+0000 7f411a000700 1 -- 172.21.15.84:0/2245774528 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4108004750 con 0x7f411c103e30 2024-04-28T12:49:50.207 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.210+0000 7f411a000700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f41000a3cd0 0x7f41000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:50.207 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.210+0000 7f411a000700 1 -- 172.21.15.84:0/2245774528 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f41080d0d00 con 0x7f411c103e30 2024-04-28T12:49:50.207 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.211+0000 7f4121200700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f41000a3cd0 0x7f41000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:50.208 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.211+0000 7f411a000700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628] conn(0x7f41000aee30 0x7f41000b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:50.208 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.211+0000 7f411a000700 1 -- 172.21.15.84:0/2245774528 --> [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41000b19a0 con 0x7f41000aee30 2024-04-28T12:49:50.208 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.211+0000 7f411a000700 1 -- 172.21.15.84:0/2245774528 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f41080d10b0 con 0x7f411c103e30 2024-04-28T12:49:50.208 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.211+0000 7f4121c00700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628] conn(0x7f41000aee30 0x7f41000b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:50.208 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.211+0000 7f4121200700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f41000a3cd0 0x7f41000a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f410c006010 tx=0x7f410c009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:50.208 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.212+0000 7f4121c00700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628] conn(0x7f41000aee30 0x7f41000b12d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:50.209 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.213+0000 7f411a000700 1 -- 172.21.15.84:0/2245774528 <== osd.3 v2:172.21.15.84:6810/3690923628 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f41000b19a0 con 0x7f41000aee30 2024-04-28T12:49:50.227 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.231+0000 7f4123c00700 1 -- 172.21.15.84:0/2245774528 --> [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f411c063fb0 con 0x7f41000aee30 2024-04-28T12:49:50.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.231+0000 7f411a000700 1 -- 172.21.15.84:0/2245774528 <== osd.3 v2:172.21.15.84:6810/3690923628 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f411c063fb0 con 0x7f41000aee30 2024-04-28T12:49:50.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.232+0000 7f4123c00700 1 -- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628] conn(0x7f41000aee30 msgr2=0x7f41000b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:50.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.232+0000 7f4123c00700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628] conn(0x7f41000aee30 0x7f41000b12d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:50.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.232+0000 7f4123c00700 1 -- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f41000a3cd0 msgr2=0x7f41000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:50.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.232+0000 7f4123c00700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f41000a3cd0 0x7f41000a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f410c006010 tx=0x7f410c009510 comp rx=0 tx=0).stop 2024-04-28T12:49:50.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.232+0000 7f4123c00700 1 -- 172.21.15.84:0/2245774528 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f411c103e30 msgr2=0x7f411c1a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:50.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.232+0000 7f4123c00700 1 --2- 172.21.15.84:0/2245774528 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f411c103e30 0x7f411c1a1f90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f410800e9f0 tx=0x7f410800ed00 comp rx=0 tx=0).stop 2024-04-28T12:49:50.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.232+0000 7f4121c00700 1 -- 172.21.15.84:0/2245774528 reap_dead start 2024-04-28T12:49:50.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.232+0000 7f4123c00700 1 -- 172.21.15.84:0/2245774528 shutdown_connections 2024-04-28T12:49:50.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.232+0000 7f4123c00700 1 -- 172.21.15.84:0/2245774528 >> 172.21.15.84:0/2245774528 conn(0x7f411c07a580 msgr2=0x7f411c10e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:50.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.232+0000 7f4123c00700 1 -- 172.21.15.84:0/2245774528 shutdown_connections 2024-04-28T12:49:50.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:50.232+0000 7f4123c00700 1 -- 172.21.15.84:0/2245774528 wait complete. 2024-04-28T12:49:51.205 INFO:teuthology.orchestra.run.smithi084.stdout:219043332148 2024-04-28T12:49:51.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-28T12:49:51.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.406+0000 7f37e2600700 1 -- 172.21.15.84:0/3182759716 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37dc06b9c0 msgr2=0x7f37dc06be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:51.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.406+0000 7f37e2600700 1 --2- 172.21.15.84:0/3182759716 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37dc06b9c0 0x7f37dc06be20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009990 tx=0x7f37c4009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.407+0000 7f37e2600700 1 -- 172.21.15.84:0/3182759716 shutdown_connections 2024-04-28T12:49:51.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.407+0000 7f37e2600700 1 --2- 172.21.15.84:0/3182759716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37dc109490 0x7f37dc109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.407+0000 7f37e2600700 1 --2- 172.21.15.84:0/3182759716 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37dc06c360 0x7f37dc1102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.407+0000 7f37e2600700 1 --2- 172.21.15.84:0/3182759716 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37dc06b9c0 0x7f37dc06be20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.407+0000 7f37e2600700 1 -- 172.21.15.84:0/3182759716 >> 172.21.15.84:0/3182759716 conn(0x7f37dc079290 msgr2=0x7f37dc0796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:51.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.407+0000 7f37e2600700 1 -- 172.21.15.84:0/3182759716 shutdown_connections 2024-04-28T12:49:51.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.407+0000 7f37e2600700 1 -- 172.21.15.84:0/3182759716 wait complete. 2024-04-28T12:49:51.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.408+0000 7f37e2600700 1 Processor -- start 2024-04-28T12:49:51.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.409+0000 7f37e2600700 1 -- start start 2024-04-28T12:49:51.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.409+0000 7f37e2600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37dc06b9c0 0x7f37dc19f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:51.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.409+0000 7f37e2600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37dc06c360 0x7f37dc19fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:51.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.409+0000 7f37e2600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37dc109490 0x7f37dc199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:51.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.409+0000 7f37e2600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f37dc1135f0 con 0x7f37dc109490 2024-04-28T12:49:51.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.409+0000 7f37e2600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f37dc113470 con 0x7f37dc06b9c0 2024-04-28T12:49:51.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.409+0000 7f37e2600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f37dc113770 con 0x7f37dc06c360 2024-04-28T12:49:51.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.410+0000 7f37dbe00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37dc109490 0x7f37dc199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:51.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.410+0000 7f37d3e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37dc06c360 0x7f37dc19fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:51.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.410+0000 7f37db400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37dc06b9c0 0x7f37dc19f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:51.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.410+0000 7f37db400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37dc06b9c0 0x7f37dc19f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:47956/0 (socket says 172.21.15.84:47956) 2024-04-28T12:49:51.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.410+0000 7f37db400700 1 -- 172.21.15.84:0/2962712843 learned_addr learned my addr 172.21.15.84:0/2962712843 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.410+0000 7f37d3e00700 1 -- 172.21.15.84:0/2962712843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37dc06b9c0 msgr2=0x7f37dc19f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.410+0000 7f37d3e00700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37dc06b9c0 0x7f37dc19f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.410+0000 7f37d3e00700 1 -- 172.21.15.84:0/2962712843 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37dc109490 msgr2=0x7f37dc199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.410+0000 7f37d3e00700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37dc109490 0x7f37dc199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.410+0000 7f37d3e00700 1 -- 172.21.15.84:0/2962712843 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37dc19a070 con 0x7f37dc06c360 2024-04-28T12:49:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.410+0000 7f37db400700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37dc06b9c0 0x7f37dc19f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.411+0000 7f37d3e00700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37dc06c360 0x7f37dc19fc90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f37c40098e0 tx=0x7f37c40057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.411+0000 7f37d8c00700 1 -- 172.21.15.84:0/2962712843 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c4027070 con 0x7f37dc06c360 2024-04-28T12:49:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.411+0000 7f37e2600700 1 -- 172.21.15.84:0/2962712843 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37dc19a300 con 0x7f37dc06c360 2024-04-28T12:49:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.411+0000 7f37d8c00700 1 -- 172.21.15.84:0/2962712843 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37c4022470 con 0x7f37dc06c360 2024-04-28T12:49:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.411+0000 7f37d8c00700 1 -- 172.21.15.84:0/2962712843 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c4017650 con 0x7f37dc06c360 2024-04-28T12:49:51.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.411+0000 7f37e2600700 1 -- 172.21.15.84:0/2962712843 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37dc1a6620 con 0x7f37dc06c360 2024-04-28T12:49:51.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.412+0000 7f37e2600700 1 -- 172.21.15.84:0/2962712843 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f37dc063bb0 con 0x7f37dc06c360 2024-04-28T12:49:51.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.414+0000 7f37d8c00700 1 -- 172.21.15.84:0/2962712843 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f37c402a070 con 0x7f37dc06c360 2024-04-28T12:49:51.412 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.416+0000 7f37d8c00700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f37c00a3d50 0x7f37c00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:51.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.416+0000 7f37d8c00700 1 -- 172.21.15.84:0/2962712843 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f37c40d1a90 con 0x7f37dc06c360 2024-04-28T12:49:51.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.416+0000 7f37db400700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f37c00a3d50 0x7f37c00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:51.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.416+0000 7f37d8c00700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251] conn(0x7f37c00aeeb0 0x7f37c00b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:51.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.416+0000 7f37d8c00700 1 -- 172.21.15.84:0/2962712843 --> [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37c00b1a20 con 0x7f37c00aeeb0 2024-04-28T12:49:51.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.416+0000 7f37d8c00700 1 -- 172.21.15.84:0/2962712843 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f37c40d1e40 con 0x7f37dc06c360 2024-04-28T12:49:51.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.417+0000 7f37dbe00700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251] conn(0x7f37c00aeeb0 0x7f37c00b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:51.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.417+0000 7f37db400700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f37c00a3d50 0x7f37c00a6210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f37dc19aa50 tx=0x7f37cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:51.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.417+0000 7f37dbe00700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251] conn(0x7f37c00aeeb0 0x7f37c00b1350 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:51.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.418+0000 7f37d8c00700 1 -- 172.21.15.84:0/2962712843 <== osd.5 v2:172.21.15.179:6808/2387861251 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37c00b1a20 con 0x7f37c00aeeb0 2024-04-28T12:49:51.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.436+0000 7f37e2600700 1 -- 172.21.15.84:0/2962712843 --> [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f37dc063fb0 con 0x7f37c00aeeb0 2024-04-28T12:49:51.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.437+0000 7f37d8c00700 1 -- 172.21.15.84:0/2962712843 <== osd.5 v2:172.21.15.179:6808/2387861251 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f37dc063fb0 con 0x7f37c00aeeb0 2024-04-28T12:49:51.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.438+0000 7f37e2600700 1 -- 172.21.15.84:0/2962712843 >> [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251] conn(0x7f37c00aeeb0 msgr2=0x7f37c00b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:51.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.438+0000 7f37e2600700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251] conn(0x7f37c00aeeb0 0x7f37c00b1350 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.438+0000 7f37e2600700 1 -- 172.21.15.84:0/2962712843 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f37c00a3d50 msgr2=0x7f37c00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:51.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.438+0000 7f37e2600700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f37c00a3d50 0x7f37c00a6210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f37dc19aa50 tx=0x7f37cc009260 comp rx=0 tx=0).stop 2024-04-28T12:49:51.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.438+0000 7f37e2600700 1 -- 172.21.15.84:0/2962712843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37dc06c360 msgr2=0x7f37dc19fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:51.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.438+0000 7f37e2600700 1 --2- 172.21.15.84:0/2962712843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37dc06c360 0x7f37dc19fc90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f37c40098e0 tx=0x7f37c40057c0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.438+0000 7f37dbe00700 1 -- 172.21.15.84:0/2962712843 reap_dead start 2024-04-28T12:49:51.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.438+0000 7f37e2600700 1 -- 172.21.15.84:0/2962712843 shutdown_connections 2024-04-28T12:49:51.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.438+0000 7f37e2600700 1 -- 172.21.15.84:0/2962712843 >> 172.21.15.84:0/2962712843 conn(0x7f37dc079290 msgr2=0x7f37dc1003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:51.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.438+0000 7f37e2600700 1 -- 172.21.15.84:0/2962712843 shutdown_connections 2024-04-28T12:49:51.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:51.438+0000 7f37e2600700 1 -- 172.21.15.84:0/2962712843 wait complete. 2024-04-28T12:49:51.525 INFO:teuthology.orchestra.run.smithi084.stdout:309237645345 2024-04-28T12:49:51.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-28T12:49:51.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.186+0000 7f5d2a600700 1 -- 172.21.15.84:0/1855233016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d2406bf00 msgr2=0x7f5d2406c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.186+0000 7f5d2a600700 1 --2- 172.21.15.84:0/1855233016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d2406bf00 0x7f5d2406c360 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009990 tx=0x7f5d10009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.186+0000 7f5d2a600700 1 -- 172.21.15.84:0/1855233016 shutdown_connections 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.186+0000 7f5d2a600700 1 --2- 172.21.15.84:0/1855233016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d24108340 0x7f5d2406b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.186+0000 7f5d2a600700 1 --2- 172.21.15.84:0/1855233016 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d24105e50 0x7f5d241062e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.186+0000 7f5d2a600700 1 --2- 172.21.15.84:0/1855233016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d2406bf00 0x7f5d2406c360 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.186+0000 7f5d2a600700 1 -- 172.21.15.84:0/1855233016 >> 172.21.15.84:0/1855233016 conn(0x7f5d24079290 msgr2=0x7f5d240796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.186+0000 7f5d2a600700 1 -- 172.21.15.84:0/1855233016 shutdown_connections 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.186+0000 7f5d2a600700 1 -- 172.21.15.84:0/1855233016 wait complete. 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.188+0000 7f5d2a600700 1 Processor -- start 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.188+0000 7f5d2a600700 1 -- start start 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.188+0000 7f5d2a600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d2406bf00 0x7f5d241a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.188+0000 7f5d2a600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d24105e50 0x7f5d241a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.188+0000 7f5d2a600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d24108340 0x7f5d241a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d2a600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5d24119c30 con 0x7f5d2406bf00 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d2a600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d24119ab0 con 0x7f5d24108340 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d2a600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5d24119db0 con 0x7f5d24105e50 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d23e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d24108340 0x7f5d241a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d22a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d24105e50 0x7f5d241a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d23400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d2406bf00 0x7f5d241a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d22a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d24105e50 0x7f5d241a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:35940/0 (socket says 172.21.15.84:35940) 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d22a00700 1 -- 172.21.15.84:0/1148318069 learned_addr learned my addr 172.21.15.84:0/1148318069 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d22a00700 1 -- 172.21.15.84:0/1148318069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d24108340 msgr2=0x7f5d241a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d22a00700 1 --2- 172.21.15.84:0/1148318069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d24108340 0x7f5d241a4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d22a00700 1 -- 172.21.15.84:0/1148318069 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d2406bf00 msgr2=0x7f5d241a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d22a00700 1 --2- 172.21.15.84:0/1148318069 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d2406bf00 0x7f5d241a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.189+0000 7f5d22a00700 1 -- 172.21.15.84:0/1148318069 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d241a5150 con 0x7f5d24105e50 2024-04-28T12:49:51.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.190+0000 7f5d22a00700 1 --2- 172.21.15.84:0/1148318069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d24105e50 0x7f5d241a0680 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5d10005910 tx=0x7f5d1000b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.190+0000 7f5d1be00700 1 -- 172.21.15.84:0/1148318069 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d10027070 con 0x7f5d24105e50 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.190+0000 7f5d1be00700 1 -- 172.21.15.84:0/1148318069 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d10022470 con 0x7f5d24105e50 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.190+0000 7f5d2a600700 1 -- 172.21.15.84:0/1148318069 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d241a53e0 con 0x7f5d24105e50 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.190+0000 7f5d1be00700 1 -- 172.21.15.84:0/1148318069 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d10017650 con 0x7f5d24105e50 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.190+0000 7f5d2a600700 1 -- 172.21.15.84:0/1148318069 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d241accc0 con 0x7f5d24105e50 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.191+0000 7f5d2a600700 1 -- 172.21.15.84:0/1148318069 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5d24105890 con 0x7f5d24105e50 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.193+0000 7f5d1be00700 1 -- 172.21.15.84:0/1148318069 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5d1002a070 con 0x7f5d24105e50 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.194+0000 7f5d1be00700 1 --2- 172.21.15.84:0/1148318069 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5d080a3cd0 0x7f5d080a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.195+0000 7f5d1be00700 1 -- 172.21.15.84:0/1148318069 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f5d100d1ac0 con 0x7f5d24105e50 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.195+0000 7f5d23400700 1 --2- 172.21.15.84:0/1148318069 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5d080a3cd0 0x7f5d080a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.195+0000 7f5d1be00700 1 --2- 172.21.15.84:0/1148318069 >> [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312] conn(0x7f5d080aee30 0x7f5d080b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.195+0000 7f5d1be00700 1 -- 172.21.15.84:0/1148318069 --> [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d100095d0 con 0x7f5d080aee30 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.195+0000 7f5d1be00700 1 -- 172.21.15.84:0/1148318069 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f5d100d1e70 con 0x7f5d24105e50 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.195+0000 7f5d23e00700 1 --2- 172.21.15.84:0/1148318069 >> [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312] conn(0x7f5d080aee30 0x7f5d080b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.195+0000 7f5d23400700 1 --2- 172.21.15.84:0/1148318069 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5d080a3cd0 0x7f5d080a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009be0 tx=0x7f5d14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:51.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.196+0000 7f5d23e00700 1 --2- 172.21.15.84:0/1148318069 >> [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312] conn(0x7f5d080aee30 0x7f5d080b12d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:51.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.197+0000 7f5d1be00700 1 -- 172.21.15.84:0/1148318069 <== osd.9 v2:172.21.15.84:6826/3448177312 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5d100095d0 con 0x7f5d080aee30 2024-04-28T12:49:51.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.215+0000 7f5d2a600700 1 -- 172.21.15.84:0/1148318069 --> [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5d24102930 con 0x7f5d080aee30 2024-04-28T12:49:51.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.215+0000 7f5d1be00700 1 -- 172.21.15.84:0/1148318069 <== osd.9 v2:172.21.15.84:6826/3448177312 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5d24102930 con 0x7f5d080aee30 2024-04-28T12:49:51.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.216+0000 7f5d2a600700 1 -- 172.21.15.84:0/1148318069 >> [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312] conn(0x7f5d080aee30 msgr2=0x7f5d080b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:51.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.216+0000 7f5d2a600700 1 --2- 172.21.15.84:0/1148318069 >> [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312] conn(0x7f5d080aee30 0x7f5d080b12d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:51.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.216+0000 7f5d2a600700 1 -- 172.21.15.84:0/1148318069 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5d080a3cd0 msgr2=0x7f5d080a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:51.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.216+0000 7f5d2a600700 1 --2- 172.21.15.84:0/1148318069 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5d080a3cd0 0x7f5d080a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009be0 tx=0x7f5d14009260 comp rx=0 tx=0).stop 2024-04-28T12:49:51.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.216+0000 7f5d2a600700 1 -- 172.21.15.84:0/1148318069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d24105e50 msgr2=0x7f5d241a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:51.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.216+0000 7f5d2a600700 1 --2- 172.21.15.84:0/1148318069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5d24105e50 0x7f5d241a0680 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5d10005910 tx=0x7f5d1000b670 comp rx=0 tx=0).stop 2024-04-28T12:49:51.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.216+0000 7f5d23e00700 1 -- 172.21.15.84:0/1148318069 reap_dead start 2024-04-28T12:49:51.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.216+0000 7f5d2a600700 1 -- 172.21.15.84:0/1148318069 shutdown_connections 2024-04-28T12:49:51.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.216+0000 7f5d2a600700 1 -- 172.21.15.84:0/1148318069 >> 172.21.15.84:0/1148318069 conn(0x7f5d24079290 msgr2=0x7f5d24104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:51.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.216+0000 7f5d2a600700 1 -- 172.21.15.84:0/1148318069 shutdown_connections 2024-04-28T12:49:51.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:48.216+0000 7f5d2a600700 1 -- 172.21.15.84:0/1148318069 wait complete. 2024-04-28T12:49:51.989 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:51 smithi084 ceph-mon[28081]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-28T12:49:51.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:51 smithi161 ceph-mon[62387]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-28T12:49:52.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:51 smithi179 ceph-mon[63029]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-28T12:49:52.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.688+0000 7fc97f200700 1 -- 172.21.15.84:0/713708097 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc978105630 msgr2=0x7fc978105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.688+0000 7fc97f200700 1 --2- 172.21.15.84:0/713708097 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc978105630 0x7fc978105ab0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc9640098f0 tx=0x7fc964009c00 comp rx=0 tx=0).stop 2024-04-28T12:49:52.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.688+0000 7fc97f200700 1 -- 172.21.15.84:0/713708097 shutdown_connections 2024-04-28T12:49:52.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.688+0000 7fc97f200700 1 --2- 172.21.15.84:0/713708097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc978105ff0 0x7fc978112500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.688+0000 7fc97f200700 1 --2- 172.21.15.84:0/713708097 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc97810b630 0x7fc97810ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.688+0000 7fc97f200700 1 --2- 172.21.15.84:0/713708097 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc978105630 0x7fc978105ab0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.688+0000 7fc97f200700 1 -- 172.21.15.84:0/713708097 >> 172.21.15.84:0/713708097 conn(0x7fc978101160 msgr2=0x7fc978103580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:52.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.689+0000 7fc97f200700 1 -- 172.21.15.84:0/713708097 shutdown_connections 2024-04-28T12:49:52.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.689+0000 7fc97f200700 1 -- 172.21.15.84:0/713708097 wait complete. 2024-04-28T12:49:52.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.690+0000 7fc97f200700 1 Processor -- start 2024-04-28T12:49:52.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.690+0000 7fc97f200700 1 -- start start 2024-04-28T12:49:52.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc97f200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc978105630 0x7fc97819bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc97f200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc978105ff0 0x7fc97819c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc97f200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97810b630 0x7fc9781a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc97f200700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc978115860 con 0x7fc978105630 2024-04-28T12:49:52.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc97f200700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc9781156e0 con 0x7fc97810b630 2024-04-28T12:49:52.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc97f200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc9781159e0 con 0x7fc978105ff0 2024-04-28T12:49:52.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc977e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc978105630 0x7fc97819bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc977400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc978105ff0 0x7fc97819c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc977e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc978105630 0x7fc97819bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:49016/0 (socket says 172.21.15.84:49016) 2024-04-28T12:49:52.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc97d200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97810b630 0x7fc9781a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc977400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc978105ff0 0x7fc97819c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:59376/0 (socket says 172.21.15.84:59376) 2024-04-28T12:49:52.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc977e00700 1 -- 172.21.15.84:0/3884502471 learned_addr learned my addr 172.21.15.84:0/3884502471 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:52.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.691+0000 7fc977400700 1 -- 172.21.15.84:0/3884502471 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97810b630 msgr2=0x7fc9781a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.692+0000 7fc977400700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc97810b630 0x7fc9781a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.692+0000 7fc977400700 1 -- 172.21.15.84:0/3884502471 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc978105630 msgr2=0x7fc97819bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.692+0000 7fc977400700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc978105630 0x7fc97819bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.692+0000 7fc977400700 1 -- 172.21.15.84:0/3884502471 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9781a0d70 con 0x7fc978105ff0 2024-04-28T12:49:52.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.692+0000 7fc977e00700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc978105630 0x7fc97819bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:52.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.692+0000 7fc977400700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc978105ff0 0x7fc97819c2f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc96400c010 tx=0x7fc96400fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:52.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.692+0000 7fc974c00700 1 -- 172.21.15.84:0/3884502471 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9640050c0 con 0x7fc978105ff0 2024-04-28T12:49:52.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.692+0000 7fc974c00700 1 -- 172.21.15.84:0/3884502471 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc96400b560 con 0x7fc978105ff0 2024-04-28T12:49:52.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.692+0000 7fc97f200700 1 -- 172.21.15.84:0/3884502471 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9781a1000 con 0x7fc978105ff0 2024-04-28T12:49:52.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.692+0000 7fc974c00700 1 -- 172.21.15.84:0/3884502471 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc964003a00 con 0x7fc978105ff0 2024-04-28T12:49:52.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.692+0000 7fc97f200700 1 -- 172.21.15.84:0/3884502471 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9781a88e0 con 0x7fc978105ff0 2024-04-28T12:49:52.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.693+0000 7fc97f200700 1 -- 172.21.15.84:0/3884502471 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc978063bb0 con 0x7fc978105ff0 2024-04-28T12:49:52.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.696+0000 7fc974c00700 1 -- 172.21.15.84:0/3884502471 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc96400b710 con 0x7fc978105ff0 2024-04-28T12:49:52.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.698+0000 7fc974c00700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc95c0a3cd0 0x7fc95c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.698+0000 7fc974c00700 1 -- 172.21.15.84:0/3884502471 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fc9640d1b00 con 0x7fc978105ff0 2024-04-28T12:49:52.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.698+0000 7fc977e00700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc95c0a3cd0 0x7fc95c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.698+0000 7fc974c00700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633] conn(0x7fc95c0aee30 0x7fc95c0b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:52.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.698+0000 7fc974c00700 1 -- 172.21.15.84:0/3884502471 --> [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc95c0b19a0 con 0x7fc95c0aee30 2024-04-28T12:49:52.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.698+0000 7fc974c00700 1 -- 172.21.15.84:0/3884502471 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fc9640d1eb0 con 0x7fc978105ff0 2024-04-28T12:49:52.696 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.699+0000 7fc97d200700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633] conn(0x7fc95c0aee30 0x7fc95c0b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:52.696 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.699+0000 7fc977e00700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc95c0a3cd0 0x7fc95c0a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc968005950 tx=0x7fc968009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:52.696 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.699+0000 7fc97d200700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633] conn(0x7fc95c0aee30 0x7fc95c0b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:52.696 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.700+0000 7fc974c00700 1 -- 172.21.15.84:0/3884502471 <== osd.1 v2:172.21.15.161:6800/449593633 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc95c0b19a0 con 0x7fc95c0aee30 2024-04-28T12:49:52.715 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.718+0000 7fc97f200700 1 -- 172.21.15.84:0/3884502471 --> [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc978064030 con 0x7fc95c0aee30 2024-04-28T12:49:52.715 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.718+0000 7fc974c00700 1 -- 172.21.15.84:0/3884502471 <== osd.1 v2:172.21.15.161:6800/449593633 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc978064030 con 0x7fc95c0aee30 2024-04-28T12:49:52.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.719+0000 7fc97f200700 1 -- 172.21.15.84:0/3884502471 >> [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633] conn(0x7fc95c0aee30 msgr2=0x7fc95c0b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.719+0000 7fc97f200700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633] conn(0x7fc95c0aee30 0x7fc95c0b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:52.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.719+0000 7fc97f200700 1 -- 172.21.15.84:0/3884502471 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc95c0a3cd0 msgr2=0x7fc95c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.719+0000 7fc97f200700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc95c0a3cd0 0x7fc95c0a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc968005950 tx=0x7fc968009310 comp rx=0 tx=0).stop 2024-04-28T12:49:52.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.719+0000 7fc97f200700 1 -- 172.21.15.84:0/3884502471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc978105ff0 msgr2=0x7fc97819c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:52.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.719+0000 7fc97f200700 1 --2- 172.21.15.84:0/3884502471 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc978105ff0 0x7fc97819c2f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc96400c010 tx=0x7fc96400fa00 comp rx=0 tx=0).stop 2024-04-28T12:49:52.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.719+0000 7fc97d200700 1 -- 172.21.15.84:0/3884502471 reap_dead start 2024-04-28T12:49:52.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.719+0000 7fc97f200700 1 -- 172.21.15.84:0/3884502471 shutdown_connections 2024-04-28T12:49:52.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.719+0000 7fc97f200700 1 -- 172.21.15.84:0/3884502471 >> 172.21.15.84:0/3884502471 conn(0x7fc978101160 msgr2=0x7fc978101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:52.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.720+0000 7fc97f200700 1 -- 172.21.15.84:0/3884502471 shutdown_connections 2024-04-28T12:49:52.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:52.720+0000 7fc97f200700 1 -- 172.21.15.84:0/3884502471 wait complete. 2024-04-28T12:49:53.252 INFO:teuthology.orchestra.run.smithi084.stdout:287762808873 2024-04-28T12:49:53.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-28T12:49:53.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:53 smithi161 ceph-mon[62387]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-28T12:49:54.047 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:53 smithi084 ceph-mon[28081]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-28T12:49:54.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:53 smithi179 ceph-mon[63029]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-28T12:49:54.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.425+0000 7f8658a00700 1 -- 172.21.15.84:0/3233745533 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8654075650 msgr2=0x7f8654075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:54.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.425+0000 7f8658a00700 1 --2- 172.21.15.84:0/3233745533 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8654075650 0x7f8654075a30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8644009990 tx=0x7f8644009ca0 comp rx=0 tx=0).stop 2024-04-28T12:49:54.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.425+0000 7f8658a00700 1 -- 172.21.15.84:0/3233745533 shutdown_connections 2024-04-28T12:49:54.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.425+0000 7f8658a00700 1 --2- 172.21.15.84:0/3233745533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f86541145f0 0x7f86541169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:54.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.425+0000 7f8658a00700 1 --2- 172.21.15.84:0/3233745533 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8654075650 0x7f8654075a30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:54.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.425+0000 7f8658a00700 1 --2- 172.21.15.84:0/3233745533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8654075f70 0x7f86541140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:54.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.425+0000 7f8658a00700 1 -- 172.21.15.84:0/3233745533 >> 172.21.15.84:0/3233745533 conn(0x7f865407a580 msgr2=0x7f865407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:54.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.425+0000 7f8658a00700 1 -- 172.21.15.84:0/3233745533 shutdown_connections 2024-04-28T12:49:54.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.425+0000 7f8658a00700 1 -- 172.21.15.84:0/3233745533 wait complete. 2024-04-28T12:49:54.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.427+0000 7f8658a00700 1 Processor -- start 2024-04-28T12:49:54.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.427+0000 7f8658a00700 1 -- start start 2024-04-28T12:49:54.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.428+0000 7f8658a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8654075650 0x7f86541a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:54.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.428+0000 7f8658a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8654075f70 0x7f86541a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:54.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.428+0000 7f8658a00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86541145f0 0x7f86541a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:54.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.428+0000 7f8658a00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8654119c90 con 0x7f8654075f70 2024-04-28T12:49:54.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.428+0000 7f8658a00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8654119b10 con 0x7f86541145f0 2024-04-28T12:49:54.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.428+0000 7f8658a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f8654119e10 con 0x7f8654075650 2024-04-28T12:49:54.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.428+0000 7f8651e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8654075650 0x7f86541a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:54.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.428+0000 7f8651e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8654075650 0x7f86541a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:59396/0 (socket says 172.21.15.84:59396) 2024-04-28T12:49:54.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.428+0000 7f8651e00700 1 -- 172.21.15.84:0/198424241 learned_addr learned my addr 172.21.15.84:0/198424241 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:54.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.428+0000 7f8651400700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8654075f70 0x7f86541a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:54.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.428+0000 7f8652800700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86541145f0 0x7f86541a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:54.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.429+0000 7f8651e00700 1 -- 172.21.15.84:0/198424241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86541145f0 msgr2=0x7f86541a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:54.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.429+0000 7f8651e00700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86541145f0 0x7f86541a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:54.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.429+0000 7f8651e00700 1 -- 172.21.15.84:0/198424241 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8654075f70 msgr2=0x7f86541a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:54.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.429+0000 7f8651e00700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8654075f70 0x7f86541a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:54.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.429+0000 7f8651e00700 1 -- 172.21.15.84:0/198424241 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86541a5170 con 0x7f8654075650 2024-04-28T12:49:54.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.429+0000 7f8651e00700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8654075650 0x7f86541a0160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f86440098e0 tx=0x7f86440057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:54.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.429+0000 7f864aa00700 1 -- 172.21.15.84:0/198424241 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864400fa90 con 0x7f8654075650 2024-04-28T12:49:54.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.429+0000 7f864aa00700 1 -- 172.21.15.84:0/198424241 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f864400fc30 con 0x7f8654075650 2024-04-28T12:49:54.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.429+0000 7f8658a00700 1 -- 172.21.15.84:0/198424241 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86541a5400 con 0x7f8654075650 2024-04-28T12:49:54.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.430+0000 7f864aa00700 1 -- 172.21.15.84:0/198424241 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864400e640 con 0x7f8654075650 2024-04-28T12:49:54.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.430+0000 7f8658a00700 1 -- 172.21.15.84:0/198424241 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86541acce0 con 0x7f8654075650 2024-04-28T12:49:54.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.430+0000 7f8658a00700 1 -- 172.21.15.84:0/198424241 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8654113ef0 con 0x7f8654075650 2024-04-28T12:49:54.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.433+0000 7f864aa00700 1 -- 172.21.15.84:0/198424241 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8644009d10 con 0x7f8654075650 2024-04-28T12:49:54.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.435+0000 7f864aa00700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f86380a3d90 0x7f86380a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:54.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.435+0000 7f864aa00700 1 -- 172.21.15.84:0/198424241 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f86440d18e0 con 0x7f8654075650 2024-04-28T12:49:54.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.435+0000 7f8651400700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f86380a3d90 0x7f86380a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:54.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.435+0000 7f864aa00700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890] conn(0x7f86380aeef0 0x7f86380b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:54.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.435+0000 7f864aa00700 1 -- 172.21.15.84:0/198424241 --> [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f864401fec0 con 0x7f86380aeef0 2024-04-28T12:49:54.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.435+0000 7f864aa00700 1 -- 172.21.15.84:0/198424241 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f86440d1c90 con 0x7f8654075650 2024-04-28T12:49:54.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.436+0000 7f8652800700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890] conn(0x7f86380aeef0 0x7f86380b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:54.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.436+0000 7f8651400700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f86380a3d90 0x7f86380a6250 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8654110fa0 tx=0x7f8640009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:54.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.436+0000 7f8652800700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890] conn(0x7f86380aeef0 0x7f86380b1390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:54.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.437+0000 7f864aa00700 1 -- 172.21.15.84:0/198424241 <== osd.11 v2:172.21.15.179:6824/3474527890 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f864401fec0 con 0x7f86380aeef0 2024-04-28T12:49:54.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.455+0000 7f8658a00700 1 -- 172.21.15.84:0/198424241 --> [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8654076d40 con 0x7f86380aeef0 2024-04-28T12:49:54.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.456+0000 7f864aa00700 1 -- 172.21.15.84:0/198424241 <== osd.11 v2:172.21.15.179:6824/3474527890 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8654076d40 con 0x7f86380aeef0 2024-04-28T12:49:54.453 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.456+0000 7f8658a00700 1 -- 172.21.15.84:0/198424241 >> [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890] conn(0x7f86380aeef0 msgr2=0x7f86380b1390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:54.453 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.456+0000 7f8658a00700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890] conn(0x7f86380aeef0 0x7f86380b1390 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:54.453 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.456+0000 7f8658a00700 1 -- 172.21.15.84:0/198424241 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f86380a3d90 msgr2=0x7f86380a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:54.453 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.456+0000 7f8658a00700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f86380a3d90 0x7f86380a6250 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8654110fa0 tx=0x7f8640009260 comp rx=0 tx=0).stop 2024-04-28T12:49:54.453 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.456+0000 7f8658a00700 1 -- 172.21.15.84:0/198424241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8654075650 msgr2=0x7f86541a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:54.453 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.456+0000 7f8658a00700 1 --2- 172.21.15.84:0/198424241 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8654075650 0x7f86541a0160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f86440098e0 tx=0x7f86440057c0 comp rx=0 tx=0).stop 2024-04-28T12:49:54.454 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.456+0000 7f8652800700 1 -- 172.21.15.84:0/198424241 reap_dead start 2024-04-28T12:49:54.454 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.456+0000 7f8658a00700 1 -- 172.21.15.84:0/198424241 shutdown_connections 2024-04-28T12:49:54.454 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.456+0000 7f8658a00700 1 -- 172.21.15.84:0/198424241 >> 172.21.15.84:0/198424241 conn(0x7f865407a580 msgr2=0x7f8654113490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:54.454 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.457+0000 7f8658a00700 1 -- 172.21.15.84:0/198424241 shutdown_connections 2024-04-28T12:49:54.454 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:54.457+0000 7f8658a00700 1 -- 172.21.15.84:0/198424241 wait complete. 2024-04-28T12:49:55.860 INFO:teuthology.orchestra.run.smithi084.stdout:163208757312 2024-04-28T12:49:55.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-28T12:49:55.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:55 smithi161 ceph-mon[62387]: pgmap v365: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:49:56.143 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:55 smithi084 ceph-mon[28081]: pgmap v365: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:49:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:55 smithi179 ceph-mon[63029]: pgmap v365: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:49:57.426 INFO:teuthology.orchestra.run.smithi084.stdout:55834574935 2024-04-28T12:49:57.426 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-28T12:49:57.856 INFO:teuthology.orchestra.run.smithi084.stdout:343597383710 2024-04-28T12:49:57.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-28T12:49:57.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:57 smithi161 ceph-mon[62387]: pgmap v366: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:49:58.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:57 smithi084 ceph-mon[28081]: pgmap v366: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:49:58.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:57 smithi179 ceph-mon[63029]: pgmap v366: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:49:58.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.145+0000 7f7410c00700 1 -- 172.21.15.84:0/3257670805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f740c076170 msgr2=0x7f740c10f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:58.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.145+0000 7f7410c00700 1 --2- 172.21.15.84:0/3257670805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f740c076170 0x7f740c10f650 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f73f80098f0 tx=0x7f73f8009c00 comp rx=0 tx=0).stop 2024-04-28T12:49:58.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.145+0000 7f7410c00700 1 -- 172.21.15.84:0/3257670805 shutdown_connections 2024-04-28T12:49:58.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.145+0000 7f7410c00700 1 --2- 172.21.15.84:0/3257670805 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f740c10fb90 0x7f740c112000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:58.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.145+0000 7f7410c00700 1 --2- 172.21.15.84:0/3257670805 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740c075850 0x7f740c075c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:58.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.145+0000 7f7410c00700 1 --2- 172.21.15.84:0/3257670805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f740c076170 0x7f740c10f650 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:58.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.145+0000 7f7410c00700 1 -- 172.21.15.84:0/3257670805 >> 172.21.15.84:0/3257670805 conn(0x7f740c0feea0 msgr2=0x7f740c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:58.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.145+0000 7f7410c00700 1 -- 172.21.15.84:0/3257670805 shutdown_connections 2024-04-28T12:49:58.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.145+0000 7f7410c00700 1 -- 172.21.15.84:0/3257670805 wait complete. 2024-04-28T12:49:58.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.147+0000 7f7410c00700 1 Processor -- start 2024-04-28T12:49:58.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.147+0000 7f7410c00700 1 -- start start 2024-04-28T12:49:58.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.147+0000 7f7410c00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740c075850 0x7f740c1a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:58.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.147+0000 7f7410c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f740c076170 0x7f740c1a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:58.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.147+0000 7f7410c00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f740c10fb90 0x7f740c19baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:58.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.147+0000 7f7410c00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f740c115750 con 0x7f740c075850 2024-04-28T12:49:58.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.147+0000 7f7410c00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f740c1155d0 con 0x7f740c10fb90 2024-04-28T12:49:58.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.147+0000 7f7410c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f740c1158d0 con 0x7f740c076170 2024-04-28T12:49:58.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.148+0000 7f740b400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740c075850 0x7f740c1a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:58.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.148+0000 7f740b400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740c075850 0x7f740c1a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:49038/0 (socket says 172.21.15.84:49038) 2024-04-28T12:49:58.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.148+0000 7f740b400700 1 -- 172.21.15.84:0/4140651283 learned_addr learned my addr 172.21.15.84:0/4140651283 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:58.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.148+0000 7f740be00700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f740c10fb90 0x7f740c19baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:58.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.148+0000 7f740aa00700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f740c076170 0x7f740c1a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:58.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.148+0000 7f740b400700 1 -- 172.21.15.84:0/4140651283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f740c076170 msgr2=0x7f740c1a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:58.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.148+0000 7f740b400700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f740c076170 0x7f740c1a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:58.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.148+0000 7f740b400700 1 -- 172.21.15.84:0/4140651283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f740c10fb90 msgr2=0x7f740c19baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:58.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.148+0000 7f740b400700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f740c10fb90 0x7f740c19baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:58.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.148+0000 7f740b400700 1 -- 172.21.15.84:0/4140651283 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f740c19c250 con 0x7f740c075850 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.148+0000 7f740be00700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f740c10fb90 0x7f740c19baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.148+0000 7f740b400700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740c075850 0x7f740c1a18e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f73fc00ec10 tx=0x7f73fc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.149+0000 7f7403e00700 1 -- 172.21.15.84:0/4140651283 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc00cc90 con 0x7f740c075850 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.149+0000 7f7410c00700 1 -- 172.21.15.84:0/4140651283 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f740c19c540 con 0x7f740c075850 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.149+0000 7f7403e00700 1 -- 172.21.15.84:0/4140651283 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73fc00ce30 con 0x7f740c075850 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.149+0000 7f7403e00700 1 -- 172.21.15.84:0/4140651283 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc010430 con 0x7f740c075850 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.149+0000 7f7410c00700 1 -- 172.21.15.84:0/4140651283 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f740c1a87b0 con 0x7f740c075850 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.150+0000 7f7410c00700 1 -- 172.21.15.84:0/4140651283 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f740c063bb0 con 0x7f740c075850 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.152+0000 7f7403e00700 1 -- 172.21.15.84:0/4140651283 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f73fc004000 con 0x7f740c075850 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.153+0000 7f7403e00700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73f00a3cd0 0x7f73f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.154+0000 7f7403e00700 1 -- 172.21.15.84:0/4140651283 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f73fc0d0de0 con 0x7f740c075850 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.154+0000 7f740aa00700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73f00a3cd0 0x7f73f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.154+0000 7f7403e00700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948] conn(0x7f73f00aee30 0x7f73f00b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.154+0000 7f7403e00700 1 -- 172.21.15.84:0/4140651283 --> [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73f00b19a0 con 0x7f73f00aee30 2024-04-28T12:49:58.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.154+0000 7f7403e00700 1 -- 172.21.15.84:0/4140651283 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f73fc0d1190 con 0x7f740c075850 2024-04-28T12:49:58.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.154+0000 7f740be00700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948] conn(0x7f73f00aee30 0x7f73f00b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:58.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.155+0000 7f740aa00700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73f00a3cd0 0x7f73f00a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009fd0 tx=0x7f73f8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:58.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.155+0000 7f740be00700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948] conn(0x7f73f00aee30 0x7f73f00b12d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:58.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.156+0000 7f7403e00700 1 -- 172.21.15.84:0/4140651283 <== osd.8 v2:172.21.15.179:6816/277418948 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73f00b19a0 con 0x7f73f00aee30 2024-04-28T12:49:58.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.174+0000 7f7410c00700 1 -- 172.21.15.84:0/4140651283 --> [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f740c064030 con 0x7f73f00aee30 2024-04-28T12:49:58.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.175+0000 7f7403e00700 1 -- 172.21.15.84:0/4140651283 <== osd.8 v2:172.21.15.179:6816/277418948 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f740c064030 con 0x7f73f00aee30 2024-04-28T12:49:58.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.175+0000 7f7410c00700 1 -- 172.21.15.84:0/4140651283 >> [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948] conn(0x7f73f00aee30 msgr2=0x7f73f00b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:58.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.175+0000 7f7410c00700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948] conn(0x7f73f00aee30 0x7f73f00b12d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:58.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.175+0000 7f7410c00700 1 -- 172.21.15.84:0/4140651283 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73f00a3cd0 msgr2=0x7f73f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:58.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.175+0000 7f7410c00700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f73f00a3cd0 0x7f73f00a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009fd0 tx=0x7f73f8019040 comp rx=0 tx=0).stop 2024-04-28T12:49:58.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.175+0000 7f7410c00700 1 -- 172.21.15.84:0/4140651283 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740c075850 msgr2=0x7f740c1a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:58.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.175+0000 7f7410c00700 1 --2- 172.21.15.84:0/4140651283 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f740c075850 0x7f740c1a18e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f73fc00ec10 tx=0x7f73fc00ef20 comp rx=0 tx=0).stop 2024-04-28T12:49:58.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.176+0000 7f740be00700 1 -- 172.21.15.84:0/4140651283 reap_dead start 2024-04-28T12:49:58.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.176+0000 7f7410c00700 1 -- 172.21.15.84:0/4140651283 shutdown_connections 2024-04-28T12:49:58.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.176+0000 7f7410c00700 1 -- 172.21.15.84:0/4140651283 >> 172.21.15.84:0/4140651283 conn(0x7f740c0feea0 msgr2=0x7f740c0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:58.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.176+0000 7f7410c00700 1 -- 172.21.15.84:0/4140651283 shutdown_connections 2024-04-28T12:49:58.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:55.176+0000 7f7410c00700 1 -- 172.21.15.84:0/4140651283 wait complete. 2024-04-28T12:49:59.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.374+0000 7fed82e00700 1 -- 172.21.15.84:0/3200208708 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed7c1094c0 msgr2=0x7fed7c1098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:59.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.374+0000 7fed82e00700 1 --2- 172.21.15.84:0/3200208708 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed7c1094c0 0x7fed7c1098a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00b0a0 tx=0x7fed6c00b3b0 comp rx=0 tx=0).stop 2024-04-28T12:49:59.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.374+0000 7fed82e00700 1 -- 172.21.15.84:0/3200208708 shutdown_connections 2024-04-28T12:49:59.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.374+0000 7fed82e00700 1 --2- 172.21.15.84:0/3200208708 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed7c103e30 0x7fed7c110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:59.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.374+0000 7fed82e00700 1 --2- 172.21.15.84:0/3200208708 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed7c1094c0 0x7fed7c1098a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:59.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.374+0000 7fed82e00700 1 --2- 172.21.15.84:0/3200208708 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed7c103470 0x7fed7c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:59.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.375+0000 7fed82e00700 1 -- 172.21.15.84:0/3200208708 >> 172.21.15.84:0/3200208708 conn(0x7fed7c07a580 msgr2=0x7fed7c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:59.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.375+0000 7fed82e00700 1 -- 172.21.15.84:0/3200208708 shutdown_connections 2024-04-28T12:49:59.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.375+0000 7fed82e00700 1 -- 172.21.15.84:0/3200208708 wait complete. 2024-04-28T12:49:59.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.376+0000 7fed82e00700 1 Processor -- start 2024-04-28T12:49:59.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.376+0000 7fed82e00700 1 -- start start 2024-04-28T12:49:59.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed82e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed7c103470 0x7fed7c19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed82e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed7c103e30 0x7fed7c19fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed82e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed7c1094c0 0x7fed7c1999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed82e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c113640 con 0x7fed7c1094c0 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed82e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c1134c0 con 0x7fed7c103e30 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed82e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c1137c0 con 0x7fed7c103470 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed80e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed7c1094c0 0x7fed7c1999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed80e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed7c1094c0 0x7fed7c1999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:49048/0 (socket says 172.21.15.84:49048) 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed80e00700 1 -- 172.21.15.84:0/689807176 learned_addr learned my addr 172.21.15.84:0/689807176 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed7b400700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed7c103e30 0x7fed7c19fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed7be00700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed7c103470 0x7fed7c19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed80e00700 1 -- 172.21.15.84:0/689807176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed7c103470 msgr2=0x7fed7c19f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed80e00700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fed7c103470 0x7fed7c19f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed80e00700 1 -- 172.21.15.84:0/689807176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed7c103e30 msgr2=0x7fed7c19fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:59.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed80e00700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed7c103e30 0x7fed7c19fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.377+0000 7fed80e00700 1 -- 172.21.15.84:0/689807176 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed7c19a100 con 0x7fed7c1094c0 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.378+0000 7fed80e00700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed7c1094c0 0x7fed7c1999a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fed7400e940 tx=0x7fed7400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.378+0000 7fed78c00700 1 -- 172.21.15.84:0/689807176 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed74004a30 con 0x7fed7c1094c0 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.378+0000 7fed78c00700 1 -- 172.21.15.84:0/689807176 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed740054d0 con 0x7fed7c1094c0 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.378+0000 7fed82e00700 1 -- 172.21.15.84:0/689807176 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed7c19a3f0 con 0x7fed7c1094c0 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.378+0000 7fed78c00700 1 -- 172.21.15.84:0/689807176 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7400ccf0 con 0x7fed7c1094c0 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.378+0000 7fed82e00700 1 -- 172.21.15.84:0/689807176 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed7c1a66b0 con 0x7fed7c1094c0 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.379+0000 7fed82e00700 1 -- 172.21.15.84:0/689807176 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fed7c063b70 con 0x7fed7c1094c0 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.381+0000 7fed78c00700 1 -- 172.21.15.84:0/689807176 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fed74005000 con 0x7fed7c1094c0 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.383+0000 7fed78c00700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed600a3d50 0x7fed600a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.383+0000 7fed78c00700 1 -- 172.21.15.84:0/689807176 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fed740d0de0 con 0x7fed7c1094c0 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.383+0000 7fed7be00700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed600a3d50 0x7fed600a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.383+0000 7fed78c00700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146] conn(0x7fed600aeeb0 0x7fed600b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:49:59.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.383+0000 7fed78c00700 1 -- 172.21.15.84:0/689807176 --> [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed7400be50 con 0x7fed600aeeb0 2024-04-28T12:49:59.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.383+0000 7fed78c00700 1 -- 172.21.15.84:0/689807176 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fed740d1190 con 0x7fed7c1094c0 2024-04-28T12:49:59.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.384+0000 7fed7b400700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146] conn(0x7fed600aeeb0 0x7fed600b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:49:59.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.384+0000 7fed7be00700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed600a3d50 0x7fed600a6210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0062a0 tx=0x7fed6c006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:59.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.384+0000 7fed7b400700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146] conn(0x7fed600aeeb0 0x7fed600b1350 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:49:59.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.385+0000 7fed78c00700 1 -- 172.21.15.84:0/689807176 <== osd.2 v2:172.21.15.179:6800/3974191146 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed7400be50 con 0x7fed600aeeb0 2024-04-28T12:49:59.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.403+0000 7fed82e00700 1 -- 172.21.15.84:0/689807176 --> [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fed7c063ff0 con 0x7fed600aeeb0 2024-04-28T12:49:59.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.404+0000 7fed78c00700 1 -- 172.21.15.84:0/689807176 <== osd.2 v2:172.21.15.179:6800/3974191146 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fed7c063ff0 con 0x7fed600aeeb0 2024-04-28T12:49:59.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.405+0000 7fed82e00700 1 -- 172.21.15.84:0/689807176 >> [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146] conn(0x7fed600aeeb0 msgr2=0x7fed600b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:59.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.405+0000 7fed82e00700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146] conn(0x7fed600aeeb0 0x7fed600b1350 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:49:59.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.405+0000 7fed82e00700 1 -- 172.21.15.84:0/689807176 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed600a3d50 msgr2=0x7fed600a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:59.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.405+0000 7fed82e00700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fed600a3d50 0x7fed600a6210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0062a0 tx=0x7fed6c006210 comp rx=0 tx=0).stop 2024-04-28T12:49:59.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.405+0000 7fed82e00700 1 -- 172.21.15.84:0/689807176 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed7c1094c0 msgr2=0x7fed7c1999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:49:59.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.405+0000 7fed82e00700 1 --2- 172.21.15.84:0/689807176 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed7c1094c0 0x7fed7c1999a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fed7400e940 tx=0x7fed7400ec50 comp rx=0 tx=0).stop 2024-04-28T12:49:59.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.405+0000 7fed80e00700 1 -- 172.21.15.84:0/689807176 reap_dead start 2024-04-28T12:49:59.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.405+0000 7fed82e00700 1 -- 172.21.15.84:0/689807176 shutdown_connections 2024-04-28T12:49:59.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.405+0000 7fed82e00700 1 -- 172.21.15.84:0/689807176 >> 172.21.15.84:0/689807176 conn(0x7fed7c07a580 msgr2=0x7fed7c1022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:49:59.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.405+0000 7fed82e00700 1 -- 172.21.15.84:0/689807176 shutdown_connections 2024-04-28T12:49:59.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:49:58.405+0000 7fed82e00700 1 -- 172.21.15.84:0/689807176 wait complete. 2024-04-28T12:49:59.845 INFO:teuthology.orchestra.run.smithi084.stdout:253403070511 2024-04-28T12:49:59.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-28T12:49:59.988 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:49:59 smithi084 ceph-mon[28081]: pgmap v367: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:49:59.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:49:59 smithi161 ceph-mon[62387]: pgmap v367: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:00.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:49:59 smithi179 ceph-mon[63029]: pgmap v367: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:00.281 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.166+0000 7f6893c00700 1 -- 172.21.15.84:0/2284674685 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f688c077950 msgr2=0x7f688c077d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:00.281 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.166+0000 7f6893c00700 1 --2- 172.21.15.84:0/2284674685 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f688c077950 0x7f688c077d30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f687c009990 tx=0x7f687c009ca0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.281 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.167+0000 7f6893c00700 1 -- 172.21.15.84:0/2284674685 shutdown_connections 2024-04-28T12:50:00.281 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.167+0000 7f6893c00700 1 --2- 172.21.15.84:0/2284674685 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f688c114300 0x7f688c1166f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.281 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.167+0000 7f6893c00700 1 --2- 172.21.15.84:0/2284674685 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f688c077950 0x7f688c077d30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.281 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.167+0000 7f6893c00700 1 --2- 172.21.15.84:0/2284674685 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f688c078270 0x7f688c113dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.281 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.167+0000 7f6893c00700 1 -- 172.21.15.84:0/2284674685 >> 172.21.15.84:0/2284674685 conn(0x7f688c101150 msgr2=0x7f688c103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.167+0000 7f6893c00700 1 -- 172.21.15.84:0/2284674685 shutdown_connections 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.167+0000 7f6893c00700 1 -- 172.21.15.84:0/2284674685 wait complete. 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.168+0000 7f6893c00700 1 Processor -- start 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.169+0000 7f6893c00700 1 -- start start 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.170+0000 7f6893c00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f688c077950 0x7f688c1a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.170+0000 7f6893c00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f688c078270 0x7f688c1a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.170+0000 7f6893c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f688c114300 0x7f688c1a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.170+0000 7f6893c00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f688c119da0 con 0x7f688c078270 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.170+0000 7f6893c00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f688c119c20 con 0x7f688c077950 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.170+0000 7f6893c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f688c119f20 con 0x7f688c114300 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.170+0000 7f6891200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f688c077950 0x7f688c1a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.170+0000 7f6891200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f688c077950 0x7f688c1a02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:49242/0 (socket says 172.21.15.84:49242) 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.170+0000 7f6891200700 1 -- 172.21.15.84:0/436894989 learned_addr learned my addr 172.21.15.84:0/436894989 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.171+0000 7f688be00700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f688c078270 0x7f688c1a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:00.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.171+0000 7f6891200700 1 -- 172.21.15.84:0/436894989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f688c114300 msgr2=0x7f688c1a4b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.171+0000 7f6891200700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f688c114300 0x7f688c1a4b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.171+0000 7f6891200700 1 -- 172.21.15.84:0/436894989 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f688c078270 msgr2=0x7f688c1a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.171+0000 7f6891200700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f688c078270 0x7f688c1a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.171+0000 7f6891200700 1 -- 172.21.15.84:0/436894989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f688c1a52c0 con 0x7f688c077950 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.171+0000 7f688be00700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f688c078270 0x7f688c1a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.171+0000 7f6891200700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f688c077950 0x7f688c1a02b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f687c0057c0 tx=0x7f687c005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.172+0000 7f6889600700 1 -- 172.21.15.84:0/436894989 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c027070 con 0x7f688c077950 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.172+0000 7f6889600700 1 -- 172.21.15.84:0/436894989 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f687c004040 con 0x7f688c077950 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.172+0000 7f6889600700 1 -- 172.21.15.84:0/436894989 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c022d80 con 0x7f688c077950 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.172+0000 7f6893c00700 1 -- 172.21.15.84:0/436894989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f688c1a5550 con 0x7f688c077950 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.172+0000 7f6893c00700 1 -- 172.21.15.84:0/436894989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f688c06ce40 con 0x7f688c077950 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.174+0000 7f6893c00700 1 -- 172.21.15.84:0/436894989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f688c1a59e0 con 0x7f688c077950 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.177+0000 7f6889600700 1 -- 172.21.15.84:0/436894989 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f687c02a070 con 0x7f688c077950 2024-04-28T12:50:00.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.179+0000 7f6889600700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f68700a3d50 0x7f68700a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:00.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.180+0000 7f6889600700 1 -- 172.21.15.84:0/436894989 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f687c0d1ce0 con 0x7f688c077950 2024-04-28T12:50:00.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.180+0000 7f6889600700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515] conn(0x7f68700aeeb0 0x7f68700b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:00.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.180+0000 7f6889600700 1 -- 172.21.15.84:0/436894989 --> [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f687c0095d0 con 0x7f68700aeeb0 2024-04-28T12:50:00.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.180+0000 7f6889600700 1 -- 172.21.15.84:0/436894989 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f687c0d2090 con 0x7f688c077950 2024-04-28T12:50:00.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.181+0000 7f6891c00700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515] conn(0x7f68700aeeb0 0x7f68700b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:00.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.181+0000 7f688be00700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f68700a3d50 0x7f68700a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:00.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.182+0000 7f6891c00700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515] conn(0x7f68700aeeb0 0x7f68700b1350 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:00.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.183+0000 7f6889600700 1 -- 172.21.15.84:0/436894989 <== osd.4 v2:172.21.15.161:6808/1673806515 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f687c0095d0 con 0x7f68700aeeb0 2024-04-28T12:50:00.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.184+0000 7f688be00700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f68700a3d50 0x7f68700a6210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f688c07a550 tx=0x7f6878008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:00.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.221+0000 7f6893c00700 1 -- 172.21.15.84:0/436894989 --> [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f688c063ed0 con 0x7f68700aeeb0 2024-04-28T12:50:00.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.223+0000 7f6889600700 1 -- 172.21.15.84:0/436894989 <== osd.4 v2:172.21.15.161:6808/1673806515 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f688c063ed0 con 0x7f68700aeeb0 2024-04-28T12:50:00.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.223+0000 7f6893c00700 1 -- 172.21.15.84:0/436894989 >> [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515] conn(0x7f68700aeeb0 msgr2=0x7f68700b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.223+0000 7f6893c00700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515] conn(0x7f68700aeeb0 0x7f68700b1350 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.223+0000 7f6893c00700 1 -- 172.21.15.84:0/436894989 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f68700a3d50 msgr2=0x7f68700a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.223+0000 7f6893c00700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f68700a3d50 0x7f68700a6210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f688c07a550 tx=0x7f6878008040 comp rx=0 tx=0).stop 2024-04-28T12:50:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.224+0000 7f6893c00700 1 -- 172.21.15.84:0/436894989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f688c077950 msgr2=0x7f688c1a02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.224+0000 7f6893c00700 1 --2- 172.21.15.84:0/436894989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f688c077950 0x7f688c1a02b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f687c0057c0 tx=0x7f687c005e50 comp rx=0 tx=0).stop 2024-04-28T12:50:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.224+0000 7f6891c00700 1 -- 172.21.15.84:0/436894989 reap_dead start 2024-04-28T12:50:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.224+0000 7f6893c00700 1 -- 172.21.15.84:0/436894989 shutdown_connections 2024-04-28T12:50:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.224+0000 7f6893c00700 1 -- 172.21.15.84:0/436894989 >> 172.21.15.84:0/436894989 conn(0x7f688c101150 msgr2=0x7f688c103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.225+0000 7f6893c00700 1 -- 172.21.15.84:0/436894989 shutdown_connections 2024-04-28T12:50:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:00.225+0000 7f6893c00700 1 -- 172.21.15.84:0/436894989 wait complete. 2024-04-28T12:50:00.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:00 smithi161 ceph-mon[62387]: overall HEALTH_OK 2024-04-28T12:50:01.001 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:00 smithi084 ceph-mon[28081]: overall HEALTH_OK 2024-04-28T12:50:01.069 INFO:teuthology.orchestra.run.smithi084.stdout:107374182476 2024-04-28T12:50:01.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-28T12:50:01.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:00 smithi179 ceph-mon[63029]: overall HEALTH_OK 2024-04-28T12:50:01.577 INFO:teuthology.orchestra.run.smithi084.stdout:77309411412 2024-04-28T12:50:01.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-28T12:50:01.892 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:01 smithi084 ceph-mon[28081]: pgmap v368: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-28T12:50:01.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:01 smithi161 ceph-mon[62387]: pgmap v368: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-28T12:50:02.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:01 smithi179 ceph-mon[63029]: pgmap v368: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-28T12:50:02.475 INFO:teuthology.orchestra.run.smithi084.stdout:141733920840 2024-04-28T12:50:02.475 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-28T12:50:02.870 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:02.876 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:02.885 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:02.897 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:02.903 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:02.910 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:02.918 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:02.922 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:03.377 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:03.985 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:03 smithi084 ceph-mon[28081]: pgmap v369: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:03.992 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:03 smithi161 ceph-mon[62387]: pgmap v369: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:04.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:03 smithi179 ceph-mon[63029]: pgmap v369: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:05.945 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:05 smithi084 ceph-mon[28081]: pgmap v370: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:05.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:05 smithi161 ceph-mon[62387]: pgmap v370: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:06.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:05 smithi179 ceph-mon[63029]: pgmap v370: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:07.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:07 smithi161 ceph-mon[62387]: pgmap v371: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:08.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:07 smithi084 ceph-mon[28081]: pgmap v371: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:08.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:07 smithi179 ceph-mon[63029]: pgmap v371: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:09.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:09 smithi084 ceph-mon[28081]: pgmap v372: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:09.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:09 smithi161 ceph-mon[62387]: pgmap v372: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:09 smithi179 ceph-mon[63029]: pgmap v372: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:11.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:11 smithi161 ceph-mon[62387]: pgmap v373: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:12.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.019+0000 7f319f000700 1 -- 172.21.15.84:0/4285740422 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3198102e60 msgr2=0x7f3198106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:12.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.019+0000 7f319f000700 1 --2- 172.21.15.84:0/4285740422 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3198102e60 0x7f3198106eb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f31840099c0 tx=0x7f3184009cd0 comp rx=0 tx=0).stop 2024-04-28T12:50:12.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.019+0000 7f319f000700 1 -- 172.21.15.84:0/4285740422 shutdown_connections 2024-04-28T12:50:12.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.019+0000 7f319f000700 1 --2- 172.21.15.84:0/4285740422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31981024b0 0x7f3198102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:12.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.019+0000 7f319f000700 1 --2- 172.21.15.84:0/4285740422 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3198107590 0x7f319810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:12.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.019+0000 7f319f000700 1 --2- 172.21.15.84:0/4285740422 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3198102e60 0x7f3198106eb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:12.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.019+0000 7f319f000700 1 -- 172.21.15.84:0/4285740422 >> 172.21.15.84:0/4285740422 conn(0x7f31980fdd40 msgr2=0x7f3198100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:12.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.019+0000 7f319f000700 1 -- 172.21.15.84:0/4285740422 shutdown_connections 2024-04-28T12:50:12.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.019+0000 7f319f000700 1 -- 172.21.15.84:0/4285740422 wait complete. 2024-04-28T12:50:12.017 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:11 smithi084 ceph-mon[28081]: pgmap v373: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:12.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.020+0000 7f319f000700 1 Processor -- start 2024-04-28T12:50:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.021+0000 7f319f000700 1 -- start start 2024-04-28T12:50:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.021+0000 7f319f000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f31981024b0 0x7f31981a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.021+0000 7f319f000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3198102e60 0x7f31981a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.021+0000 7f319f000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3198107590 0x7f319819ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.021+0000 7f319f000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3198112450 con 0x7f31981024b0 2024-04-28T12:50:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.021+0000 7f319f000700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f31981122d0 con 0x7f3198102e60 2024-04-28T12:50:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.021+0000 7f319f000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f31981125d0 con 0x7f3198107590 2024-04-28T12:50:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.021+0000 7f319dc00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f31981024b0 0x7f31981a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.021+0000 7f319dc00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f31981024b0 0x7f31981a18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:45936/0 (socket says 172.21.15.84:45936) 2024-04-28T12:50:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.021+0000 7f319dc00700 1 -- 172.21.15.84:0/636304984 learned_addr learned my addr 172.21.15.84:0/636304984 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.021+0000 7f319d200700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3198102e60 0x7f31981a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.021+0000 7f319e600700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3198107590 0x7f319819ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:12.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.022+0000 7f319dc00700 1 -- 172.21.15.84:0/636304984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3198107590 msgr2=0x7f319819ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:12.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.022+0000 7f319dc00700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3198107590 0x7f319819ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:12.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.022+0000 7f319dc00700 1 -- 172.21.15.84:0/636304984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3198102e60 msgr2=0x7f31981a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:12.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.022+0000 7f319dc00700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3198102e60 0x7f31981a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:12.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.022+0000 7f319dc00700 1 -- 172.21.15.84:0/636304984 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319819c1d0 con 0x7f31981024b0 2024-04-28T12:50:12.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.022+0000 7f319dc00700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f31981024b0 0x7f31981a18b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f318c00e9a0 tx=0x7f318c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:12.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.022+0000 7f3196000700 1 -- 172.21.15.84:0/636304984 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c00ca10 con 0x7f31981024b0 2024-04-28T12:50:12.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.023+0000 7f3196000700 1 -- 172.21.15.84:0/636304984 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318c00cbb0 con 0x7f31981024b0 2024-04-28T12:50:12.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.023+0000 7f319f000700 1 -- 172.21.15.84:0/636304984 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319819c4c0 con 0x7f31981024b0 2024-04-28T12:50:12.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.023+0000 7f3196000700 1 -- 172.21.15.84:0/636304984 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c018680 con 0x7f31981024b0 2024-04-28T12:50:12.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.023+0000 7f319f000700 1 -- 172.21.15.84:0/636304984 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31981a8780 con 0x7f31981024b0 2024-04-28T12:50:12.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.024+0000 7f319f000700 1 -- 172.21.15.84:0/636304984 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f319806a110 con 0x7f31981024b0 2024-04-28T12:50:12.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.026+0000 7f3196000700 1 -- 172.21.15.84:0/636304984 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f318c010430 con 0x7f31981024b0 2024-04-28T12:50:12.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.027+0000 7f3196000700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f31800a3d50 0x7f31800a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:12.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.027+0000 7f3196000700 1 -- 172.21.15.84:0/636304984 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f318c0d0a80 con 0x7f31981024b0 2024-04-28T12:50:12.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.027+0000 7f319d200700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f31800a3d50 0x7f31800a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:12.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.028+0000 7f319d200700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f31800a3d50 0x7f31800a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3184000c00 tx=0x7f3184005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:12.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.032+0000 7f3196000700 1 -- 172.21.15.84:0/636304984 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f318c0d2e20 con 0x7f31981024b0 2024-04-28T12:50:12.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:11 smithi179 ceph-mon[63029]: pgmap v373: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:12.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.263+0000 7f319f000700 1 -- 172.21.15.84:0/636304984 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f319802e490 con 0x7f31981024b0 2024-04-28T12:50:12.260 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.264+0000 7f3196000700 1 -- 172.21.15.84:0/636304984 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f318c09cf40 con 0x7f31981024b0 2024-04-28T12:50:12.261 INFO:teuthology.orchestra.run.smithi084.stdout:197568495678 2024-04-28T12:50:12.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.267+0000 7f319f000700 1 -- 172.21.15.84:0/636304984 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f31800a3d50 msgr2=0x7f31800a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:12.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.267+0000 7f319f000700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f31800a3d50 0x7f31800a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3184000c00 tx=0x7f3184005fb0 comp rx=0 tx=0).stop 2024-04-28T12:50:12.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.267+0000 7f319f000700 1 -- 172.21.15.84:0/636304984 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f31981024b0 msgr2=0x7f31981a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:12.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.267+0000 7f319f000700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f31981024b0 0x7f31981a18b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f318c00e9a0 tx=0x7f318c00ecb0 comp rx=0 tx=0).stop 2024-04-28T12:50:12.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.267+0000 7f319f000700 1 -- 172.21.15.84:0/636304984 shutdown_connections 2024-04-28T12:50:12.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.267+0000 7f319f000700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3198107590 0x7f319819ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:12.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.267+0000 7f319f000700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f31981024b0 0x7f31981a18b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:12.265 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.267+0000 7f319f000700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f31800a3d50 0x7f31800a6210 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:12.265 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.267+0000 7f319f000700 1 --2- 172.21.15.84:0/636304984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3198102e60 0x7f31981a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:12.265 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.267+0000 7f319f000700 1 -- 172.21.15.84:0/636304984 >> 172.21.15.84:0/636304984 conn(0x7f31980fdd40 msgr2=0x7f319810e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:12.265 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.267+0000 7f319f000700 1 -- 172.21.15.84:0/636304984 shutdown_connections 2024-04-28T12:50:12.265 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:12.267+0000 7f319f000700 1 -- 172.21.15.84:0/636304984 wait complete. 2024-04-28T12:50:12.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:12 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/636304984' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-28T12:50:13.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:12 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/636304984' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-28T12:50:13.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:12 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/636304984' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-28T12:50:13.743 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495672 got 197568495678 for osd.6 2024-04-28T12:50:13.743 DEBUG:teuthology.parallel:result is None 2024-04-28T12:50:14.105 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:13 smithi084 ceph-mon[28081]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:14.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:13 smithi179 ceph-mon[63029]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:13 smithi161 ceph-mon[62387]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:14.344 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:15.108 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.111+0000 7fe6dbe00700 1 -- 172.21.15.84:0/4190557266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d410fa50 msgr2=0x7fe6d410fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:15.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.111+0000 7fe6dbe00700 1 --2- 172.21.15.84:0/4190557266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d410fa50 0x7fe6d410fe30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009990 tx=0x7fe6c8009ca0 comp rx=0 tx=0).stop 2024-04-28T12:50:15.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.112+0000 7fe6dbe00700 1 -- 172.21.15.84:0/4190557266 shutdown_connections 2024-04-28T12:50:15.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.112+0000 7fe6dbe00700 1 --2- 172.21.15.84:0/4190557266 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d410a410 0x7fe6d4116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:15.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.112+0000 7fe6dbe00700 1 --2- 172.21.15.84:0/4190557266 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d410fa50 0x7fe6d410fe30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:15.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.112+0000 7fe6dbe00700 1 --2- 172.21.15.84:0/4190557266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d4109a50 0x7fe6d4109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:15.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.112+0000 7fe6dbe00700 1 -- 172.21.15.84:0/4190557266 >> 172.21.15.84:0/4190557266 conn(0x7fe6d4076bf0 msgr2=0x7fe6d4108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:15.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.112+0000 7fe6dbe00700 1 -- 172.21.15.84:0/4190557266 shutdown_connections 2024-04-28T12:50:15.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.112+0000 7fe6dbe00700 1 -- 172.21.15.84:0/4190557266 wait complete. 2024-04-28T12:50:15.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.113+0000 7fe6dbe00700 1 Processor -- start 2024-04-28T12:50:15.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.113+0000 7fe6dbe00700 1 -- start start 2024-04-28T12:50:15.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.113+0000 7fe6dbe00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d4109a50 0x7fe6d41a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:15.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.113+0000 7fe6dbe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d410a410 0x7fe6d41a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:15.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.113+0000 7fe6dbe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d410fa50 0x7fe6d41a4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:15.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.113+0000 7fe6dbe00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d4119ca0 con 0x7fe6d4109a50 2024-04-28T12:50:15.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.113+0000 7fe6dbe00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d4119b20 con 0x7fe6d410fa50 2024-04-28T12:50:15.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.113+0000 7fe6dbe00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d4119e20 con 0x7fe6d410a410 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.114+0000 7fe6d9400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d4109a50 0x7fe6d41a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.114+0000 7fe6d9e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d410fa50 0x7fe6d41a4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.114+0000 7fe6d9e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d410fa50 0x7fe6d41a4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:35150/0 (socket says 172.21.15.84:35150) 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.114+0000 7fe6d9e00700 1 -- 172.21.15.84:0/2379036043 learned_addr learned my addr 172.21.15.84:0/2379036043 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.114+0000 7fe6d8a00700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d410a410 0x7fe6d41a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.114+0000 7fe6d9e00700 1 -- 172.21.15.84:0/2379036043 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d410a410 msgr2=0x7fe6d41a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.114+0000 7fe6d9e00700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d410a410 0x7fe6d41a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.114+0000 7fe6d9e00700 1 -- 172.21.15.84:0/2379036043 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d4109a50 msgr2=0x7fe6d41a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.114+0000 7fe6d9e00700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d4109a50 0x7fe6d41a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.114+0000 7fe6d9e00700 1 -- 172.21.15.84:0/2379036043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d41a5140 con 0x7fe6d410fa50 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.114+0000 7fe6d9400700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d4109a50 0x7fe6d41a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.114+0000 7fe6d8a00700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d410a410 0x7fe6d41a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.115+0000 7fe6d9e00700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d410fa50 0x7fe6d41a4a00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc007ee0 tx=0x7fe6cc00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:15.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.115+0000 7fe6d2000700 1 -- 172.21.15.84:0/2379036043 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc00dd20 con 0x7fe6d410fa50 2024-04-28T12:50:15.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.115+0000 7fe6d2000700 1 -- 172.21.15.84:0/2379036043 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6cc004020 con 0x7fe6d410fa50 2024-04-28T12:50:15.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.115+0000 7fe6dbe00700 1 -- 172.21.15.84:0/2379036043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d41a5430 con 0x7fe6d410fa50 2024-04-28T12:50:15.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.115+0000 7fe6d2000700 1 -- 172.21.15.84:0/2379036043 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc015660 con 0x7fe6d410fa50 2024-04-28T12:50:15.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.115+0000 7fe6dbe00700 1 -- 172.21.15.84:0/2379036043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d41accb0 con 0x7fe6d410fa50 2024-04-28T12:50:15.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.116+0000 7fe6dbe00700 1 -- 172.21.15.84:0/2379036043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d406a110 con 0x7fe6d410fa50 2024-04-28T12:50:15.114 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.118+0000 7fe6d2000700 1 -- 172.21.15.84:0/2379036043 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe6cc0041d0 con 0x7fe6d410fa50 2024-04-28T12:50:15.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.120+0000 7fe6d2000700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe6bc0a3cd0 0x7fe6bc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:15.116 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.120+0000 7fe6d2000700 1 -- 172.21.15.84:0/2379036043 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fe6cc0d03a0 con 0x7fe6d410fa50 2024-04-28T12:50:15.116 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.120+0000 7fe6d9400700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe6bc0a3cd0 0x7fe6bc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:15.116 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.120+0000 7fe6d9400700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe6bc0a3cd0 0x7fe6bc0a6190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8006010 tx=0x7fe6c800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:15.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.122+0000 7fe6d2000700 1 -- 172.21.15.84:0/2379036043 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe6cc010070 con 0x7fe6d410fa50 2024-04-28T12:50:15.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.361+0000 7fe6dbe00700 1 -- 172.21.15.84:0/2379036043 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fe6d402e490 con 0x7fe6d410fa50 2024-04-28T12:50:15.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.361+0000 7fe6d2000700 1 -- 172.21.15.84:0/2379036043 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fe6cc09c8c0 con 0x7fe6d410fa50 2024-04-28T12:50:15.358 INFO:teuthology.orchestra.run.smithi084.stdout:309237645350 2024-04-28T12:50:15.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.365+0000 7fe6dbe00700 1 -- 172.21.15.84:0/2379036043 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe6bc0a3cd0 msgr2=0x7fe6bc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:15.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.365+0000 7fe6dbe00700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe6bc0a3cd0 0x7fe6bc0a6190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8006010 tx=0x7fe6c800b260 comp rx=0 tx=0).stop 2024-04-28T12:50:15.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.365+0000 7fe6dbe00700 1 -- 172.21.15.84:0/2379036043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d410fa50 msgr2=0x7fe6d41a4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:15.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.365+0000 7fe6dbe00700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d410fa50 0x7fe6d41a4a00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc007ee0 tx=0x7fe6cc00d3b0 comp rx=0 tx=0).stop 2024-04-28T12:50:15.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.365+0000 7fe6dbe00700 1 -- 172.21.15.84:0/2379036043 shutdown_connections 2024-04-28T12:50:15.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.365+0000 7fe6dbe00700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe6bc0a3cd0 0x7fe6bc0a6190 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:15.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.365+0000 7fe6dbe00700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d410fa50 0x7fe6d41a4a00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:15.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.365+0000 7fe6dbe00700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe6d4109a50 0x7fe6d41a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:15.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.365+0000 7fe6dbe00700 1 --2- 172.21.15.84:0/2379036043 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe6d410a410 0x7fe6d41a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:15.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.365+0000 7fe6dbe00700 1 -- 172.21.15.84:0/2379036043 >> 172.21.15.84:0/2379036043 conn(0x7fe6d4076bf0 msgr2=0x7fe6d4077210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:15.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.365+0000 7fe6dbe00700 1 -- 172.21.15.84:0/2379036043 shutdown_connections 2024-04-28T12:50:15.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.366+0000 7fe6dbe00700 1 -- 172.21.15.84:0/2379036043 wait complete. 2024-04-28T12:50:15.637 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:15 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2379036043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-28T12:50:15.665 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:15 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2379036043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-28T12:50:15.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:15 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2379036043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-28T12:50:16.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:16 smithi161 ceph-mon[62387]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:16.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:16 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/780425099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-28T12:50:16.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.807+0000 7faff8a00700 1 -- 172.21.15.84:0/1861826138 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faff4108680 msgr2=0x7faff4110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.807+0000 7faff8a00700 1 --2- 172.21.15.84:0/1861826138 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faff4108680 0x7faff4110250 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fafec00b0d0 tx=0x7fafec00b3e0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.807+0000 7faff8a00700 1 -- 172.21.15.84:0/1861826138 shutdown_connections 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.807+0000 7faff8a00700 1 --2- 172.21.15.84:0/1861826138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faff41035a0 0x7faff4103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.807+0000 7faff8a00700 1 --2- 172.21.15.84:0/1861826138 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faff4108680 0x7faff4110250 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.807+0000 7faff8a00700 1 --2- 172.21.15.84:0/1861826138 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faff4103f50 0x7faff4107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.807+0000 7faff8a00700 1 -- 172.21.15.84:0/1861826138 >> 172.21.15.84:0/1861826138 conn(0x7faff40fee30 msgr2=0x7faff4101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.808+0000 7faff8a00700 1 -- 172.21.15.84:0/1861826138 shutdown_connections 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.808+0000 7faff8a00700 1 -- 172.21.15.84:0/1861826138 wait complete. 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.809+0000 7faff8a00700 1 Processor -- start 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.809+0000 7faff8a00700 1 -- start start 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff8a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faff41035a0 0x7faff419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff8a00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faff4103f50 0x7faff419fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff8a00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faff4108680 0x7faff4199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:16.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff8a00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7faff4113520 con 0x7faff41035a0 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff8a00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faff41133a0 con 0x7faff4103f50 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff8a00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7faff41136a0 con 0x7faff4108680 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff3400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faff41035a0 0x7faff419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff3400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faff41035a0 0x7faff419f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:45978/0 (socket says 172.21.15.84:45978) 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff3e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faff4108680 0x7faff4199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff3400700 1 -- 172.21.15.84:0/780425099 learned_addr learned my addr 172.21.15.84:0/780425099 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff2a00700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faff4103f50 0x7faff419fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff3e00700 1 -- 172.21.15.84:0/780425099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faff4103f50 msgr2=0x7faff419fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff3e00700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faff4103f50 0x7faff419fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff3e00700 1 -- 172.21.15.84:0/780425099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faff41035a0 msgr2=0x7faff419f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff3e00700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faff41035a0 0x7faff419f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.810+0000 7faff3e00700 1 -- 172.21.15.84:0/780425099 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff4199fe0 con 0x7faff4108680 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.811+0000 7faff3e00700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faff4108680 0x7faff4199800 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fafec00b0d0 tx=0x7fafec007a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:16.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.811+0000 7fafebe00700 1 -- 172.21.15.84:0/780425099 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec01f070 con 0x7faff4108680 2024-04-28T12:50:16.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.811+0000 7faff8a00700 1 -- 172.21.15.84:0/780425099 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faff419a270 con 0x7faff4108680 2024-04-28T12:50:16.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.811+0000 7fafebe00700 1 -- 172.21.15.84:0/780425099 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafec004c50 con 0x7faff4108680 2024-04-28T12:50:16.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.811+0000 7fafebe00700 1 -- 172.21.15.84:0/780425099 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec01b430 con 0x7faff4108680 2024-04-28T12:50:16.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.811+0000 7faff8a00700 1 -- 172.21.15.84:0/780425099 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faff41a6490 con 0x7faff4108680 2024-04-28T12:50:16.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.812+0000 7faff8a00700 1 -- 172.21.15.84:0/780425099 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faff4104a80 con 0x7faff4108680 2024-04-28T12:50:16.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.815+0000 7fafebe00700 1 -- 172.21.15.84:0/780425099 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fafec022070 con 0x7faff4108680 2024-04-28T12:50:16.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.816+0000 7fafebe00700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fafd80a3d50 0x7fafd80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:16.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.816+0000 7fafebe00700 1 -- 172.21.15.84:0/780425099 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fafec00f020 con 0x7faff4108680 2024-04-28T12:50:16.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.816+0000 7faff3400700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fafd80a3d50 0x7fafd80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:16.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.817+0000 7faff3400700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fafd80a3d50 0x7fafd80a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009d20 tx=0x7fafe4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:16.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:15.818+0000 7fafebe00700 1 -- 172.21.15.84:0/780425099 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fafec029020 con 0x7faff4108680 2024-04-28T12:50:16.896 INFO:teuthology.orchestra.run.smithi084.stdout:163208757316 2024-04-28T12:50:16.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.051+0000 7faff8a00700 1 -- 172.21.15.84:0/780425099 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7faff419ace0 con 0x7faff4108680 2024-04-28T12:50:16.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.052+0000 7fafebe00700 1 -- 172.21.15.84:0/780425099 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fafec09e0a0 con 0x7faff4108680 2024-04-28T12:50:16.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.055+0000 7faff8a00700 1 -- 172.21.15.84:0/780425099 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fafd80a3d50 msgr2=0x7fafd80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:16.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.055+0000 7faff8a00700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fafd80a3d50 0x7fafd80a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009d20 tx=0x7fafe4009260 comp rx=0 tx=0).stop 2024-04-28T12:50:16.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.055+0000 7faff8a00700 1 -- 172.21.15.84:0/780425099 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faff4108680 msgr2=0x7faff4199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:16.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.055+0000 7faff8a00700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faff4108680 0x7faff4199800 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fafec00b0d0 tx=0x7fafec007a00 comp rx=0 tx=0).stop 2024-04-28T12:50:16.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.055+0000 7faff8a00700 1 -- 172.21.15.84:0/780425099 shutdown_connections 2024-04-28T12:50:16.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.055+0000 7faff8a00700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faff4108680 0x7faff4199800 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.055+0000 7faff8a00700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faff41035a0 0x7faff419f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.055+0000 7faff8a00700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fafd80a3d50 0x7fafd80a6210 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.055+0000 7faff8a00700 1 --2- 172.21.15.84:0/780425099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faff4103f50 0x7faff419fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:16.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.055+0000 7faff8a00700 1 -- 172.21.15.84:0/780425099 >> 172.21.15.84:0/780425099 conn(0x7faff40fee30 msgr2=0x7faff4101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:16.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.056+0000 7faff8a00700 1 -- 172.21.15.84:0/780425099 shutdown_connections 2024-04-28T12:50:16.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:16.056+0000 7faff8a00700 1 -- 172.21.15.84:0/780425099 wait complete. 2024-04-28T12:50:16.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:16 smithi084 ceph-mon[28081]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:16.897 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:16 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/780425099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-28T12:50:16.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:16 smithi179 ceph-mon[63029]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:16.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:16 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/780425099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-28T12:50:17.849 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757312 got 163208757316 for osd.5 2024-04-28T12:50:17.849 DEBUG:teuthology.parallel:result is None 2024-04-28T12:50:17.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:17 smithi161 ceph-mon[62387]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:18.076 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:17 smithi084 ceph-mon[28081]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:18.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:17 smithi179 ceph-mon[63029]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:18.978 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:19.605 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.136+0000 7fae54200700 1 -- 172.21.15.84:0/2835129123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4c077950 msgr2=0x7fae4c077d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:19.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.136+0000 7fae54200700 1 --2- 172.21.15.84:0/2835129123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4c077950 0x7fae4c077d30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00b0a0 tx=0x7fae3c00b3b0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.136+0000 7fae54200700 1 -- 172.21.15.84:0/2835129123 shutdown_connections 2024-04-28T12:50:19.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.136+0000 7fae54200700 1 --2- 172.21.15.84:0/2835129123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae4c1142d0 0x7fae4c1166c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.136+0000 7fae54200700 1 --2- 172.21.15.84:0/2835129123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4c077950 0x7fae4c077d30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.136+0000 7fae54200700 1 --2- 172.21.15.84:0/2835129123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae4c078270 0x7fae4c113d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.136+0000 7fae54200700 1 -- 172.21.15.84:0/2835129123 >> 172.21.15.84:0/2835129123 conn(0x7fae4c101120 msgr2=0x7fae4c103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:19.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.136+0000 7fae54200700 1 -- 172.21.15.84:0/2835129123 shutdown_connections 2024-04-28T12:50:19.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.137+0000 7fae54200700 1 -- 172.21.15.84:0/2835129123 wait complete. 2024-04-28T12:50:19.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.138+0000 7fae54200700 1 Processor -- start 2024-04-28T12:50:19.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.138+0000 7fae54200700 1 -- start start 2024-04-28T12:50:19.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.139+0000 7fae54200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae4c077950 0x7fae4c1a0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:19.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.139+0000 7fae54200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4c078270 0x7fae4c1a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.139+0000 7fae54200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae4c1142d0 0x7fae4c1a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.139+0000 7fae54200700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c119cc0 con 0x7fae4c077950 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.139+0000 7fae54200700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c119b40 con 0x7fae4c1142d0 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.139+0000 7fae54200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fae4c119e40 con 0x7fae4c078270 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.139+0000 7fae51800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae4c077950 0x7fae4c1a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.139+0000 7fae51800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae4c077950 0x7fae4c1a0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50920/0 (socket says 172.21.15.84:50920) 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.139+0000 7fae51800700 1 -- 172.21.15.84:0/2442286876 learned_addr learned my addr 172.21.15.84:0/2442286876 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.139+0000 7fae50e00700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4c078270 0x7fae4c1a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.139+0000 7fae52200700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae4c1142d0 0x7fae4c1a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.140+0000 7fae51800700 1 -- 172.21.15.84:0/2442286876 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4c078270 msgr2=0x7fae4c1a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.140+0000 7fae51800700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4c078270 0x7fae4c1a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.140+0000 7fae51800700 1 -- 172.21.15.84:0/2442286876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae4c1142d0 msgr2=0x7fae4c1a4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.140+0000 7fae51800700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae4c1142d0 0x7fae4c1a4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.140+0000 7fae51800700 1 -- 172.21.15.84:0/2442286876 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae4c1a5210 con 0x7fae4c077950 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.140+0000 7fae51800700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae4c077950 0x7fae4c1a0250 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00d010 tx=0x7fae3c004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:19.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.140+0000 7fae4a000700 1 -- 172.21.15.84:0/2442286876 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c021070 con 0x7fae4c077950 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.140+0000 7fae4a000700 1 -- 172.21.15.84:0/2442286876 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae3c007e30 con 0x7fae4c077950 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.140+0000 7fae54200700 1 -- 172.21.15.84:0/2442286876 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae4c1a54a0 con 0x7fae4c077950 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.140+0000 7fae4a000700 1 -- 172.21.15.84:0/2442286876 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3c004700 con 0x7fae4c077950 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.140+0000 7fae54200700 1 -- 172.21.15.84:0/2442286876 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae4c1acd80 con 0x7fae4c077950 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.142+0000 7fae54200700 1 -- 172.21.15.84:0/2442286876 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae4c06a110 con 0x7fae4c077950 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.143+0000 7fae4a000700 1 -- 172.21.15.84:0/2442286876 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fae3c007980 con 0x7fae4c077950 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.145+0000 7fae4a000700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fae300a3cd0 0x7fae300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.145+0000 7fae4a000700 1 -- 172.21.15.84:0/2442286876 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fae3c00f020 con 0x7fae4c077950 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.145+0000 7fae50e00700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fae300a3cd0 0x7fae300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.146+0000 7fae50e00700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fae300a3cd0 0x7fae300a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae4c07a600 tx=0x7fae38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.150+0000 7fae4a000700 1 -- 172.21.15.84:0/2442286876 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fae3c02a020 con 0x7fae4c077950 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.383+0000 7fae54200700 1 -- 172.21.15.84:0/2442286876 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fae4c02e490 con 0x7fae4c077950 2024-04-28T12:50:19.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.383+0000 7fae4a000700 1 -- 172.21.15.84:0/2442286876 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fae3c003c00 con 0x7fae4c077950 2024-04-28T12:50:19.609 INFO:teuthology.orchestra.run.smithi084.stdout:219043332154 2024-04-28T12:50:19.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.386+0000 7fae54200700 1 -- 172.21.15.84:0/2442286876 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fae300a3cd0 msgr2=0x7fae300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:19.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.386+0000 7fae54200700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fae300a3cd0 0x7fae300a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae4c07a600 tx=0x7fae38009260 comp rx=0 tx=0).stop 2024-04-28T12:50:19.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.386+0000 7fae54200700 1 -- 172.21.15.84:0/2442286876 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae4c077950 msgr2=0x7fae4c1a0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:19.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.386+0000 7fae54200700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae4c077950 0x7fae4c1a0250 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00d010 tx=0x7fae3c004d00 comp rx=0 tx=0).stop 2024-04-28T12:50:19.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.386+0000 7fae54200700 1 -- 172.21.15.84:0/2442286876 shutdown_connections 2024-04-28T12:50:19.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.386+0000 7fae54200700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fae300a3cd0 0x7fae300a6190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.386+0000 7fae54200700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae4c1142d0 0x7fae4c1a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.386+0000 7fae54200700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae4c077950 0x7fae4c1a0250 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.386+0000 7fae54200700 1 --2- 172.21.15.84:0/2442286876 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fae4c078270 0x7fae4c1a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:19.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.386+0000 7fae54200700 1 -- 172.21.15.84:0/2442286876 >> 172.21.15.84:0/2442286876 conn(0x7fae4c101120 msgr2=0x7fae4c103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:19.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.387+0000 7fae54200700 1 -- 172.21.15.84:0/2442286876 shutdown_connections 2024-04-28T12:50:19.611 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:19.387+0000 7fae54200700 1 -- 172.21.15.84:0/2442286876 wait complete. 2024-04-28T12:50:19.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:19 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2442286876' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-28T12:50:19.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:19 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2442286876' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-28T12:50:19.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:19 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2442286876' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-28T12:50:20.667 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:20 smithi084 ceph-mon[28081]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:20.742 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:20 smithi161 ceph-mon[62387]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:20.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:20 smithi179 ceph-mon[63029]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:21.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:21 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3883692901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-28T12:50:21.854 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:21 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3883692901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-28T12:50:21.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:21 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3883692901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-28T12:50:22.400 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.362+0000 7f54a4600700 1 -- 172.21.15.84:0/1402401572 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f549c075650 msgr2=0x7f549c075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:22.400 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.362+0000 7f54a4600700 1 --2- 172.21.15.84:0/1402401572 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f549c075650 0x7f549c075a30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5490009990 tx=0x7f5490009ca0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.400 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.363+0000 7f54a4600700 1 -- 172.21.15.84:0/1402401572 shutdown_connections 2024-04-28T12:50:22.400 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.363+0000 7f54a4600700 1 --2- 172.21.15.84:0/1402401572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f549c10fe20 0x7f549c112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.400 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.363+0000 7f54a4600700 1 --2- 172.21.15.84:0/1402401572 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f549c075650 0x7f549c075a30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.363+0000 7f54a4600700 1 --2- 172.21.15.84:0/1402401572 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f549c075f70 0x7f549c10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.363+0000 7f54a4600700 1 -- 172.21.15.84:0/1402401572 >> 172.21.15.84:0/1402401572 conn(0x7f549c07a580 msgr2=0x7f549c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.363+0000 7f54a4600700 1 -- 172.21.15.84:0/1402401572 shutdown_connections 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.363+0000 7f54a4600700 1 -- 172.21.15.84:0/1402401572 wait complete. 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.364+0000 7f54a4600700 1 Processor -- start 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.364+0000 7f54a4600700 1 -- start start 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.364+0000 7f54a4600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f549c075650 0x7f549c1a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.364+0000 7f54a4600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f549c075f70 0x7f549c1a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.364+0000 7f54a4600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f549c10fe20 0x7f549c19ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.364+0000 7f54a4600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f549c1157d0 con 0x7f549c075650 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.364+0000 7f54a4600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f549c115650 con 0x7f549c075f70 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.364+0000 7f54a4600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f549c115950 con 0x7f549c10fe20 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.365+0000 7f54a1c00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f549c075650 0x7f549c1a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.365+0000 7f54a1c00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f549c075650 0x7f549c1a18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50966/0 (socket says 172.21.15.84:50966) 2024-04-28T12:50:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.365+0000 7f54a1c00700 1 -- 172.21.15.84:0/2939140257 learned_addr learned my addr 172.21.15.84:0/2939140257 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.365+0000 7f54a1200700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f549c075f70 0x7f549c1a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.365+0000 7f54a2600700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f549c10fe20 0x7f549c19ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.365+0000 7f54a1c00700 1 -- 172.21.15.84:0/2939140257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f549c10fe20 msgr2=0x7f549c19ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.365+0000 7f54a1c00700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f549c10fe20 0x7f549c19ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.365+0000 7f54a1c00700 1 -- 172.21.15.84:0/2939140257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f549c075f70 msgr2=0x7f549c1a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.365+0000 7f54a1c00700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f549c075f70 0x7f549c1a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.365+0000 7f54a1c00700 1 -- 172.21.15.84:0/2939140257 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549c19c1d0 con 0x7f549c075650 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.365+0000 7f54a1200700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f549c075f70 0x7f549c1a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.365+0000 7f54a1c00700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f549c075650 0x7f549c1a18b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f54900057f0 tx=0x7f549000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.366+0000 7f549a000700 1 -- 172.21.15.84:0/2939140257 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490026070 con 0x7f549c075650 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.366+0000 7f54a4600700 1 -- 172.21.15.84:0/2939140257 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f549c19c460 con 0x7f549c075650 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.366+0000 7f549a000700 1 -- 172.21.15.84:0/2939140257 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5490021410 con 0x7f549c075650 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.366+0000 7f549a000700 1 -- 172.21.15.84:0/2939140257 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490017650 con 0x7f549c075650 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.366+0000 7f54a4600700 1 -- 172.21.15.84:0/2939140257 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f549c1a8780 con 0x7f549c075650 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.367+0000 7f54a4600700 1 -- 172.21.15.84:0/2939140257 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f549c06a110 con 0x7f549c075650 2024-04-28T12:50:22.402 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.369+0000 7f549a000700 1 -- 172.21.15.84:0/2939140257 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5490029070 con 0x7f549c075650 2024-04-28T12:50:22.403 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.370+0000 7f549a000700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f54840a3d50 0x7f54840a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:22.403 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.370+0000 7f549a000700 1 -- 172.21.15.84:0/2939140257 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f54900d14a0 con 0x7f549c075650 2024-04-28T12:50:22.403 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.371+0000 7f54a1200700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f54840a3d50 0x7f54840a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:22.403 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.371+0000 7f54a1200700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f54840a3d50 0x7f54840a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f549c111ae0 tx=0x7f5488009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:22.403 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.379+0000 7f549a000700 1 -- 172.21.15.84:0/2939140257 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f549005e060 con 0x7f549c075650 2024-04-28T12:50:22.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.614+0000 7f54a4600700 1 -- 172.21.15.84:0/2939140257 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f549c02e490 con 0x7f549c075650 2024-04-28T12:50:22.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.614+0000 7f549a000700 1 -- 172.21.15.84:0/2939140257 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5490013070 con 0x7f549c075650 2024-04-28T12:50:22.611 INFO:teuthology.orchestra.run.smithi084.stdout:253403070516 2024-04-28T12:50:22.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.617+0000 7f54a4600700 1 -- 172.21.15.84:0/2939140257 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f54840a3d50 msgr2=0x7f54840a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:22.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.617+0000 7f54a4600700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f54840a3d50 0x7f54840a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f549c111ae0 tx=0x7f5488009260 comp rx=0 tx=0).stop 2024-04-28T12:50:22.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.617+0000 7f54a4600700 1 -- 172.21.15.84:0/2939140257 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f549c075650 msgr2=0x7f549c1a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:22.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.617+0000 7f54a4600700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f549c075650 0x7f549c1a18b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f54900057f0 tx=0x7f549000bfd0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.617+0000 7f54a4600700 1 -- 172.21.15.84:0/2939140257 shutdown_connections 2024-04-28T12:50:22.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.617+0000 7f54a4600700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f549c10fe20 0x7f549c19ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.617+0000 7f54a4600700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f549c075650 0x7f549c1a18b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.617+0000 7f54a4600700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f54840a3d50 0x7f54840a6210 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.617+0000 7f54a4600700 1 --2- 172.21.15.84:0/2939140257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f549c075f70 0x7f549c1a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.617+0000 7f54a4600700 1 -- 172.21.15.84:0/2939140257 >> 172.21.15.84:0/2939140257 conn(0x7f549c07a580 msgr2=0x7f549c1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:22.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.617+0000 7f54a4600700 1 -- 172.21.15.84:0/2939140257 shutdown_connections 2024-04-28T12:50:22.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:22.618+0000 7f54a4600700 1 -- 172.21.15.84:0/2939140257 wait complete. 2024-04-28T12:50:22.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:22 smithi161 ceph-mon[62387]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:22.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:22 smithi084 ceph-mon[28081]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:22.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.110+0000 7f5907200700 1 -- 172.21.15.84:0/3177535733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5900075650 msgr2=0x7f5900075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:22.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.110+0000 7f5907200700 1 --2- 172.21.15.84:0/3177535733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5900075650 0x7f5900075a30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009990 tx=0x7f58f0009ca0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.110+0000 7f5907200700 1 -- 172.21.15.84:0/3177535733 shutdown_connections 2024-04-28T12:50:22.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.110+0000 7f5907200700 1 --2- 172.21.15.84:0/3177535733 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59001145f0 0x7f59001169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.110+0000 7f5907200700 1 --2- 172.21.15.84:0/3177535733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5900075650 0x7f5900075a30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.110+0000 7f5907200700 1 --2- 172.21.15.84:0/3177535733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5900075f70 0x7f59001140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.110+0000 7f5907200700 1 -- 172.21.15.84:0/3177535733 >> 172.21.15.84:0/3177535733 conn(0x7f590007a580 msgr2=0x7f590007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:22.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.110+0000 7f5907200700 1 -- 172.21.15.84:0/3177535733 shutdown_connections 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.110+0000 7f5907200700 1 -- 172.21.15.84:0/3177535733 wait complete. 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.111+0000 7f5907200700 1 Processor -- start 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.112+0000 7f5907200700 1 -- start start 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.112+0000 7f5907200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5900075650 0x7f59001a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.112+0000 7f5907200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5900075f70 0x7f59001a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.112+0000 7f5907200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59001145f0 0x7f59001a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.112+0000 7f5907200700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5900119cc0 con 0x7f59001145f0 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.112+0000 7f5907200700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5900119b40 con 0x7f5900075650 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.112+0000 7f5907200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5900119e40 con 0x7f5900075f70 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.112+0000 7f5905200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59001145f0 0x7f59001a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.112+0000 7f58ff400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5900075f70 0x7f59001a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.112+0000 7f58ffe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5900075650 0x7f59001a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.113+0000 7f58ffe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5900075650 0x7f59001a0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:59008/0 (socket says 172.21.15.84:59008) 2024-04-28T12:50:22.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.112+0000 7f5905200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59001145f0 0x7f59001a4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50946/0 (socket says 172.21.15.84:50946) 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.113+0000 7f58ffe00700 1 -- 172.21.15.84:0/3883692901 learned_addr learned my addr 172.21.15.84:0/3883692901 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.113+0000 7f58ffe00700 1 -- 172.21.15.84:0/3883692901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5900075f70 msgr2=0x7f59001a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.113+0000 7f58ffe00700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5900075f70 0x7f59001a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.113+0000 7f58ffe00700 1 -- 172.21.15.84:0/3883692901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59001145f0 msgr2=0x7f59001a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.113+0000 7f58ffe00700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59001145f0 0x7f59001a4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.113+0000 7f58ffe00700 1 -- 172.21.15.84:0/3883692901 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59001a5220 con 0x7f5900075650 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.113+0000 7f5905200700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59001145f0 0x7f59001a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.113+0000 7f58ffe00700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5900075650 0x7f59001a0210 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f58f000b2e0 tx=0x7f58f00057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.114+0000 7f58fcc00700 1 -- 172.21.15.84:0/3883692901 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0026070 con 0x7f5900075650 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.114+0000 7f58fcc00700 1 -- 172.21.15.84:0/3883692901 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58f0021410 con 0x7f5900075650 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.114+0000 7f5907200700 1 -- 172.21.15.84:0/3883692901 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59001a54b0 con 0x7f5900075650 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.114+0000 7f58fcc00700 1 -- 172.21.15.84:0/3883692901 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0017650 con 0x7f5900075650 2024-04-28T12:50:22.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.114+0000 7f5907200700 1 -- 172.21.15.84:0/3883692901 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59001acd90 con 0x7f5900075650 2024-04-28T12:50:22.882 INFO:teuthology.orchestra.run.smithi084.stdout:343597383715 2024-04-28T12:50:22.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.115+0000 7f5907200700 1 -- 172.21.15.84:0/3883692901 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5900076d40 con 0x7f5900075650 2024-04-28T12:50:22.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.117+0000 7f58fcc00700 1 -- 172.21.15.84:0/3883692901 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f58f0029070 con 0x7f5900075650 2024-04-28T12:50:22.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.118+0000 7f58fcc00700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f58e40a3cd0 0x7f58e40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:22.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.119+0000 7f58fcc00700 1 -- 172.21.15.84:0/3883692901 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f58f00d19c0 con 0x7f5900075650 2024-04-28T12:50:22.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.119+0000 7f58ff400700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f58e40a3cd0 0x7f58e40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:22.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.119+0000 7f58ff400700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f58e40a3cd0 0x7f58e40a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5900101b90 tx=0x7f58ec009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.121+0000 7f58fcc00700 1 -- 172.21.15.84:0/3883692901 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f58f0030080 con 0x7f5900075650 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.377+0000 7f5907200700 1 -- 172.21.15.84:0/3883692901 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f590006a110 con 0x7f5900075650 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.378+0000 7f58fcc00700 1 -- 172.21.15.84:0/3883692901 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f58f0013070 con 0x7f5900075650 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.381+0000 7f5907200700 1 -- 172.21.15.84:0/3883692901 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f58e40a3cd0 msgr2=0x7f58e40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.381+0000 7f5907200700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f58e40a3cd0 0x7f58e40a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5900101b90 tx=0x7f58ec009190 comp rx=0 tx=0).stop 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.381+0000 7f5907200700 1 -- 172.21.15.84:0/3883692901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5900075650 msgr2=0x7f59001a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.381+0000 7f5907200700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5900075650 0x7f59001a0210 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f58f000b2e0 tx=0x7f58f00057c0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.381+0000 7f5907200700 1 -- 172.21.15.84:0/3883692901 shutdown_connections 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.381+0000 7f5907200700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f58e40a3cd0 0x7f58e40a6190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.381+0000 7f5907200700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5900075650 0x7f59001a0210 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.381+0000 7f5907200700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59001145f0 0x7f59001a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.381+0000 7f5907200700 1 --2- 172.21.15.84:0/3883692901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5900075f70 0x7f59001a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.381+0000 7f5907200700 1 -- 172.21.15.84:0/3883692901 >> 172.21.15.84:0/3883692901 conn(0x7f590007a580 msgr2=0x7f5900110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:22.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.382+0000 7f5907200700 1 -- 172.21.15.84:0/3883692901 shutdown_connections 2024-04-28T12:50:22.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:21.382+0000 7f5907200700 1 -- 172.21.15.84:0/3883692901 wait complete. 2024-04-28T12:50:22.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:22 smithi179 ceph-mon[63029]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:23.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:23 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2939140257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-28T12:50:23.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:23 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2939140257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-28T12:50:23.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:23 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2939140257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-28T12:50:24.723 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:24 smithi084 ceph-mon[28081]: pgmap v379: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:24.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:24 smithi161 ceph-mon[62387]: pgmap v379: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:24.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:24 smithi179 ceph-mon[63029]: pgmap v379: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:25.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:25 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:50:25.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:25 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:50:25.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:25 smithi084 ceph-mon[28081]: pgmap v380: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:25.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.896+0000 7f0981e00700 1 -- 172.21.15.84:0/4153758114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f097c103e30 msgr2=0x7f097c110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:25.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.896+0000 7f0981e00700 1 --2- 172.21.15.84:0/4153758114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f097c103e30 0x7f097c110290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f097400b0a0 tx=0x7f097400b3b0 comp rx=0 tx=0).stop 2024-04-28T12:50:25.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.896+0000 7f0981e00700 1 -- 172.21.15.84:0/4153758114 shutdown_connections 2024-04-28T12:50:25.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.896+0000 7f0981e00700 1 --2- 172.21.15.84:0/4153758114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f097c103e30 0x7f097c110290 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:25.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.896+0000 7f0981e00700 1 --2- 172.21.15.84:0/4153758114 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f097c1094c0 0x7f097c1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:25.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.896+0000 7f0981e00700 1 --2- 172.21.15.84:0/4153758114 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f097c103470 0x7f097c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:25.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.896+0000 7f0981e00700 1 -- 172.21.15.84:0/4153758114 >> 172.21.15.84:0/4153758114 conn(0x7f097c07a580 msgr2=0x7f097c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:25.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.896+0000 7f0981e00700 1 -- 172.21.15.84:0/4153758114 shutdown_connections 2024-04-28T12:50:25.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.896+0000 7f0981e00700 1 -- 172.21.15.84:0/4153758114 wait complete. 2024-04-28T12:50:25.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.897+0000 7f0981e00700 1 Processor -- start 2024-04-28T12:50:25.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.897+0000 7f0981e00700 1 -- start start 2024-04-28T12:50:25.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.898+0000 7f0981e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f097c103470 0x7f097c075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:25.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.898+0000 7f0981e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f097c103e30 0x7f097c075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:25.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.898+0000 7f0981e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f097c1094c0 0x7f097c0761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:25.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.898+0000 7f0981e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f097c113610 con 0x7f097c1094c0 2024-04-28T12:50:25.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.898+0000 7f0981e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f097c113490 con 0x7f097c103470 2024-04-28T12:50:25.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.898+0000 7f0981e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f097c113790 con 0x7f097c103e30 2024-04-28T12:50:25.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.898+0000 7f097aa00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f097c103470 0x7f097c075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:25.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.898+0000 7f097a000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f097c103e30 0x7f097c075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:25.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.898+0000 7f097aa00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f097c103470 0x7f097c075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:59064/0 (socket says 172.21.15.84:59064) 2024-04-28T12:50:25.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.898+0000 7f097aa00700 1 -- 172.21.15.84:0/2060550093 learned_addr learned my addr 172.21.15.84:0/2060550093 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:25.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.898+0000 7f097b400700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f097c1094c0 0x7f097c0761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:25.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.899+0000 7f097b400700 1 -- 172.21.15.84:0/2060550093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f097c103e30 msgr2=0x7f097c075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:25.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.899+0000 7f097b400700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f097c103e30 0x7f097c075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:25.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.899+0000 7f097b400700 1 -- 172.21.15.84:0/2060550093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f097c103470 msgr2=0x7f097c075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:25.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.899+0000 7f097b400700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f097c103470 0x7f097c075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:25.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.899+0000 7f097b400700 1 -- 172.21.15.84:0/2060550093 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097c1a8030 con 0x7f097c1094c0 2024-04-28T12:50:25.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.899+0000 7f097aa00700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f097c103470 0x7f097c075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:25.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.900+0000 7f097b400700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f097c1094c0 0x7f097c0761a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f097400b070 tx=0x7f0974004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:25.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.900+0000 7f0980a00700 1 -- 172.21.15.84:0/2060550093 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974021070 con 0x7f097c1094c0 2024-04-28T12:50:25.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.900+0000 7f0981e00700 1 -- 172.21.15.84:0/2060550093 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097c1a82c0 con 0x7f097c1094c0 2024-04-28T12:50:25.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.900+0000 7f0981e00700 1 -- 172.21.15.84:0/2060550093 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f097c1a8800 con 0x7f097c1094c0 2024-04-28T12:50:25.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.901+0000 7f0980a00700 1 -- 172.21.15.84:0/2060550093 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0974007e30 con 0x7f097c1094c0 2024-04-28T12:50:25.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.901+0000 7f0980a00700 1 -- 172.21.15.84:0/2060550093 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097402a060 con 0x7f097c1094c0 2024-04-28T12:50:25.898 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.901+0000 7f0981e00700 1 -- 172.21.15.84:0/2060550093 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f095c005330 con 0x7f097c1094c0 2024-04-28T12:50:25.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.904+0000 7f0980a00700 1 -- 172.21.15.84:0/2060550093 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f097401d450 con 0x7f097c1094c0 2024-04-28T12:50:25.901 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.905+0000 7f0980a00700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f09600a3d50 0x7f09600a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:25.901 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.906+0000 7f0980a00700 1 -- 172.21.15.84:0/2060550093 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f09740d1760 con 0x7f097c1094c0 2024-04-28T12:50:25.902 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.906+0000 7f097aa00700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f09600a3d50 0x7f09600a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:25.902 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.907+0000 7f097aa00700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f09600a3d50 0x7f09600a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f096c011550 tx=0x7f096c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:25.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.913+0000 7f0980a00700 1 -- 172.21.15.84:0/2060550093 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f097409dcb0 con 0x7f097c1094c0 2024-04-28T12:50:26.149 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.153+0000 7f0981e00700 1 -- 172.21.15.84:0/2060550093 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f095c0058f0 con 0x7f097c1094c0 2024-04-28T12:50:26.150 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.154+0000 7f0980a00700 1 -- 172.21.15.84:0/2060550093 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f097409d5b0 con 0x7f097c1094c0 2024-04-28T12:50:26.150 INFO:teuthology.orchestra.run.smithi084.stdout:55834574941 2024-04-28T12:50:26.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.156+0000 7f0981e00700 1 -- 172.21.15.84:0/2060550093 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f09600a3d50 msgr2=0x7f09600a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:26.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.156+0000 7f0981e00700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f09600a3d50 0x7f09600a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f096c011550 tx=0x7f096c009260 comp rx=0 tx=0).stop 2024-04-28T12:50:26.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.156+0000 7f0981e00700 1 -- 172.21.15.84:0/2060550093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f097c1094c0 msgr2=0x7f097c0761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:26.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.156+0000 7f0981e00700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f097c1094c0 0x7f097c0761a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f097400b070 tx=0x7f0974004c80 comp rx=0 tx=0).stop 2024-04-28T12:50:26.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.156+0000 7f0981e00700 1 -- 172.21.15.84:0/2060550093 shutdown_connections 2024-04-28T12:50:26.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.156+0000 7f0981e00700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f09600a3d50 0x7f09600a6210 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:26.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.156+0000 7f0981e00700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f097c103470 0x7f097c075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:26.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.156+0000 7f0981e00700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f097c1094c0 0x7f097c0761a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:26.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.156+0000 7f0981e00700 1 --2- 172.21.15.84:0/2060550093 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f097c103e30 0x7f097c075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:26.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.156+0000 7f0981e00700 1 -- 172.21.15.84:0/2060550093 >> 172.21.15.84:0/2060550093 conn(0x7f097c07a580 msgr2=0x7f097c102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:26.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.157+0000 7f0981e00700 1 -- 172.21.15.84:0/2060550093 shutdown_connections 2024-04-28T12:50:26.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:26.157+0000 7f0981e00700 1 -- 172.21.15.84:0/2060550093 wait complete. 2024-04-28T12:50:26.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:50:26.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:25 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:50:26.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:25 smithi179 ceph-mon[63029]: pgmap v380: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:26.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:25 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:50:26.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:25 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:50:26.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:25 smithi161 ceph-mon[62387]: pgmap v380: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:26.883 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332148 got 219043332154 for osd.7 2024-04-28T12:50:26.884 DEBUG:teuthology.parallel:result is None 2024-04-28T12:50:26.992 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:26 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2010132778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-28T12:50:26.992 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:50:26.992 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:50:26.992 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:50:26.992 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:26 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2060550093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-28T12:50:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:26 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2010132778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-28T12:50:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:50:27.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:50:27.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:50:27.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:26 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2060550093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-28T12:50:27.213 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.737+0000 7f3973200700 1 -- 172.21.15.84:0/3437567910 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f396c06bad0 msgr2=0x7f396c1102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.737+0000 7f3973200700 1 --2- 172.21.15.84:0/3437567910 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f396c06bad0 0x7f396c1102e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f396400b0a0 tx=0x7f396400b3b0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.737+0000 7f3973200700 1 -- 172.21.15.84:0/3437567910 shutdown_connections 2024-04-28T12:50:27.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.737+0000 7f3973200700 1 --2- 172.21.15.84:0/3437567910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c1094a0 0x7f396c109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.737+0000 7f3973200700 1 --2- 172.21.15.84:0/3437567910 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f396c06bad0 0x7f396c1102e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.737+0000 7f3973200700 1 --2- 172.21.15.84:0/3437567910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f396c06b130 0x7f396c06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.737+0000 7f3973200700 1 -- 172.21.15.84:0/3437567910 >> 172.21.15.84:0/3437567910 conn(0x7f396c0792d0 msgr2=0x7f396c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:27.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.738+0000 7f3973200700 1 -- 172.21.15.84:0/3437567910 shutdown_connections 2024-04-28T12:50:27.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.738+0000 7f3973200700 1 -- 172.21.15.84:0/3437567910 wait complete. 2024-04-28T12:50:27.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.739+0000 7f3973200700 1 Processor -- start 2024-04-28T12:50:27.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.739+0000 7f3973200700 1 -- start start 2024-04-28T12:50:27.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.739+0000 7f3973200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f396c06b130 0x7f396c19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:27.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.739+0000 7f3973200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f396c06bad0 0x7f396c19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.739+0000 7f3973200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c1094a0 0x7f396c1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.739+0000 7f3973200700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f396c113620 con 0x7f396c06b130 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.739+0000 7f3973200700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f396c1134a0 con 0x7f396c1094a0 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.739+0000 7f3973200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f396c1137a0 con 0x7f396c06bad0 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.739+0000 7f396be00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f396c06b130 0x7f396c19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.740+0000 7f396be00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f396c06b130 0x7f396c19f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50992/0 (socket says 172.21.15.84:50992) 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.740+0000 7f396be00700 1 -- 172.21.15.84:0/2010132778 learned_addr learned my addr 172.21.15.84:0/2010132778 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.740+0000 7f3971200700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c1094a0 0x7f396c1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.740+0000 7f396b400700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f396c06bad0 0x7f396c19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.740+0000 7f396be00700 1 -- 172.21.15.84:0/2010132778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f396c06bad0 msgr2=0x7f396c19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.740+0000 7f396be00700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f396c06bad0 0x7f396c19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.740+0000 7f396be00700 1 -- 172.21.15.84:0/2010132778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c1094a0 msgr2=0x7f396c1998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.740+0000 7f396be00700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c1094a0 0x7f396c1998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.740+0000 7f396be00700 1 -- 172.21.15.84:0/2010132778 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c19a020 con 0x7f396c06b130 2024-04-28T12:50:27.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.740+0000 7f396be00700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f396c06b130 0x7f396c19f700 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f395c00e9a0 tx=0x7f395c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:27.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.741+0000 7f3968c00700 1 -- 172.21.15.84:0/2010132778 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c00cbd0 con 0x7f396c06b130 2024-04-28T12:50:27.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.741+0000 7f3968c00700 1 -- 172.21.15.84:0/2010132778 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f395c00cd70 con 0x7f396c06b130 2024-04-28T12:50:27.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.741+0000 7f3973200700 1 -- 172.21.15.84:0/2010132778 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396c19a310 con 0x7f396c06b130 2024-04-28T12:50:27.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.741+0000 7f3968c00700 1 -- 172.21.15.84:0/2010132778 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c018680 con 0x7f396c06b130 2024-04-28T12:50:27.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.741+0000 7f3973200700 1 -- 172.21.15.84:0/2010132778 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396c06ce40 con 0x7f396c06b130 2024-04-28T12:50:27.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.743+0000 7f3973200700 1 -- 172.21.15.84:0/2010132778 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396c06a110 con 0x7f396c06b130 2024-04-28T12:50:27.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.745+0000 7f3968c00700 1 -- 172.21.15.84:0/2010132778 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f395c010970 con 0x7f396c06b130 2024-04-28T12:50:27.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.746+0000 7f3968c00700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f39500a3cd0 0x7f39500a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:27.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.746+0000 7f3968c00700 1 -- 172.21.15.84:0/2010132778 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f395c0d0c60 con 0x7f396c06b130 2024-04-28T12:50:27.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.747+0000 7f396b400700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f39500a3cd0 0x7f39500a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:27.217 INFO:teuthology.orchestra.run.smithi084.stdout:34359738466 2024-04-28T12:50:27.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.747+0000 7f396b400700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f39500a3cd0 0x7f39500a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f396c079840 tx=0x7f3958009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:27.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.751+0000 7f3968c00700 1 -- 172.21.15.84:0/2010132778 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f395c09d120 con 0x7f396c06b130 2024-04-28T12:50:27.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.995+0000 7f3973200700 1 -- 172.21.15.84:0/2010132778 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f396c02e490 con 0x7f396c06b130 2024-04-28T12:50:27.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.995+0000 7f3968c00700 1 -- 172.21.15.84:0/2010132778 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f395c09ca20 con 0x7f396c06b130 2024-04-28T12:50:27.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.998+0000 7f3973200700 1 -- 172.21.15.84:0/2010132778 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f39500a3cd0 msgr2=0x7f39500a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.998+0000 7f3973200700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f39500a3cd0 0x7f39500a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f396c079840 tx=0x7f3958009260 comp rx=0 tx=0).stop 2024-04-28T12:50:27.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.998+0000 7f3973200700 1 -- 172.21.15.84:0/2010132778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f396c06b130 msgr2=0x7f396c19f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.998+0000 7f3973200700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f396c06b130 0x7f396c19f700 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f395c00e9a0 tx=0x7f395c00ed60 comp rx=0 tx=0).stop 2024-04-28T12:50:27.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.999+0000 7f3973200700 1 -- 172.21.15.84:0/2010132778 shutdown_connections 2024-04-28T12:50:27.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.999+0000 7f3973200700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f39500a3cd0 0x7f39500a6190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.999+0000 7f3973200700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c1094a0 0x7f396c1998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.999+0000 7f3973200700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f396c06b130 0x7f396c19f700 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.999+0000 7f3973200700 1 --2- 172.21.15.84:0/2010132778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f396c06bad0 0x7f396c19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.999+0000 7f3973200700 1 -- 172.21.15.84:0/2010132778 >> 172.21.15.84:0/2010132778 conn(0x7f396c0792d0 msgr2=0x7f396c100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:27.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.999+0000 7f3973200700 1 -- 172.21.15.84:0/2010132778 shutdown_connections 2024-04-28T12:50:27.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:25.999+0000 7f3973200700 1 -- 172.21.15.84:0/2010132778 wait complete. 2024-04-28T12:50:27.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:26 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2010132778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-28T12:50:27.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:50:27.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:50:27.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:50:27.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:26 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2060550093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-28T12:50:27.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.724+0000 7f7cca000700 1 -- 172.21.15.84:0/1518512261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cc4108390 msgr2=0x7f7cc4108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.724+0000 7f7cca000700 1 --2- 172.21.15.84:0/1518512261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cc4108390 0x7f7cc4108770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40098f0 tx=0x7f7cb4009c00 comp rx=0 tx=0).stop 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.724+0000 7f7cca000700 1 -- 172.21.15.84:0/1518512261 shutdown_connections 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.724+0000 7f7cca000700 1 --2- 172.21.15.84:0/1518512261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cc4102ce0 0x7f7cc410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.724+0000 7f7cca000700 1 --2- 172.21.15.84:0/1518512261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cc4108390 0x7f7cc4108770 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.724+0000 7f7cca000700 1 --2- 172.21.15.84:0/1518512261 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc4102320 0x7f7cc41027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.724+0000 7f7cca000700 1 -- 172.21.15.84:0/1518512261 >> 172.21.15.84:0/1518512261 conn(0x7f7cc40792d0 msgr2=0x7f7cc40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.724+0000 7f7cca000700 1 -- 172.21.15.84:0/1518512261 shutdown_connections 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.724+0000 7f7cca000700 1 -- 172.21.15.84:0/1518512261 wait complete. 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.725+0000 7f7cca000700 1 Processor -- start 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.725+0000 7f7cca000700 1 -- start start 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.726+0000 7f7cca000700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cc4102320 0x7f7cc419f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.726+0000 7f7cca000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc4102ce0 0x7f7cc419fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.726+0000 7f7cca000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cc4108390 0x7f7cc4199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.726+0000 7f7cca000700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc41124b0 con 0x7f7cc4102ce0 2024-04-28T12:50:27.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.726+0000 7f7cca000700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc4112330 con 0x7f7cc4108390 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.726+0000 7f7cca000700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc4112630 con 0x7f7cc4102320 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.726+0000 7f7cc3400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cc4102320 0x7f7cc419f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.726+0000 7f7cc3e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cc4108390 0x7f7cc4199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.726+0000 7f7cc3400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cc4102320 0x7f7cc419f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:34254/0 (socket says 172.21.15.84:34254) 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.726+0000 7f7cc3400700 1 -- 172.21.15.84:0/3069968517 learned_addr learned my addr 172.21.15.84:0/3069968517 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.726+0000 7f7cc2a00700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc4102ce0 0x7f7cc419fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.727+0000 7f7cc3400700 1 -- 172.21.15.84:0/3069968517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cc4108390 msgr2=0x7f7cc4199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.727+0000 7f7cc3400700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cc4108390 0x7f7cc4199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.727+0000 7f7cc3400700 1 -- 172.21.15.84:0/3069968517 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc4102ce0 msgr2=0x7f7cc419fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.727+0000 7f7cc3400700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc4102ce0 0x7f7cc419fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.727+0000 7f7cc3400700 1 -- 172.21.15.84:0/3069968517 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc419a1a0 con 0x7f7cc4102320 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.727+0000 7f7cc3400700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cc4102320 0x7f7cc419f830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7cb400f990 tx=0x7f7cb400f9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.727+0000 7f7cbbe00700 1 -- 172.21.15.84:0/3069968517 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4027070 con 0x7f7cc4102320 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.727+0000 7f7cbbe00700 1 -- 172.21.15.84:0/3069968517 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cb4005570 con 0x7f7cc4102320 2024-04-28T12:50:27.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.727+0000 7f7cca000700 1 -- 172.21.15.84:0/3069968517 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc419a430 con 0x7f7cc4102320 2024-04-28T12:50:27.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.727+0000 7f7cbbe00700 1 -- 172.21.15.84:0/3069968517 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4004930 con 0x7f7cc4102320 2024-04-28T12:50:27.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.728+0000 7f7cca000700 1 -- 172.21.15.84:0/3069968517 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc41091a0 con 0x7f7cc4102320 2024-04-28T12:50:27.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.729+0000 7f7cca000700 1 -- 172.21.15.84:0/3069968517 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cc406a110 con 0x7f7cc4102320 2024-04-28T12:50:27.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.731+0000 7f7cbbe00700 1 -- 172.21.15.84:0/3069968517 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7cb402a070 con 0x7f7cc4102320 2024-04-28T12:50:27.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.732+0000 7f7cbbe00700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ca80a3d50 0x7f7ca80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:27.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.733+0000 7f7cbbe00700 1 -- 172.21.15.84:0/3069968517 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f7cb40d18f0 con 0x7f7cc4102320 2024-04-28T12:50:27.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.733+0000 7f7cc2a00700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ca80a3d50 0x7f7ca80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:27.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.733+0000 7f7cc2a00700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ca80a3d50 0x7f7ca80a6210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40ff6d0 tx=0x7f7cb0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:27.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.735+0000 7f7cbbe00700 1 -- 172.21.15.84:0/3069968517 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7cb4031070 con 0x7f7cc4102320 2024-04-28T12:50:27.807 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645345 got 309237645350 for osd.10 2024-04-28T12:50:27.807 DEBUG:teuthology.parallel:result is None 2024-04-28T12:50:27.967 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.970+0000 7f7cca000700 1 -- 172.21.15.84:0/3069968517 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f7cc410a580 con 0x7f7cc4102320 2024-04-28T12:50:27.967 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.971+0000 7f7cbbe00700 1 -- 172.21.15.84:0/3069968517 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7cb4013030 con 0x7f7cc4102320 2024-04-28T12:50:27.968 INFO:teuthology.orchestra.run.smithi084.stdout:77309411418 2024-04-28T12:50:27.970 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.974+0000 7f7cca000700 1 -- 172.21.15.84:0/3069968517 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ca80a3d50 msgr2=0x7f7ca80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.974+0000 7f7cca000700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ca80a3d50 0x7f7ca80a6210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40ff6d0 tx=0x7f7cb0009310 comp rx=0 tx=0).stop 2024-04-28T12:50:27.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.974+0000 7f7cca000700 1 -- 172.21.15.84:0/3069968517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cc4102320 msgr2=0x7f7cc419f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:27.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.974+0000 7f7cca000700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cc4102320 0x7f7cc419f830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7cb400f990 tx=0x7f7cb400f9c0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.974+0000 7f7cca000700 1 -- 172.21.15.84:0/3069968517 shutdown_connections 2024-04-28T12:50:27.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.974+0000 7f7cca000700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ca80a3d50 0x7f7ca80a6210 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.974+0000 7f7cca000700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cc4108390 0x7f7cc4199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.974+0000 7f7cca000700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7cc4102320 0x7f7cc419f830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.974+0000 7f7cca000700 1 --2- 172.21.15.84:0/3069968517 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc4102ce0 0x7f7cc419fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:27.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.974+0000 7f7cca000700 1 -- 172.21.15.84:0/3069968517 >> 172.21.15.84:0/3069968517 conn(0x7f7cc40792d0 msgr2=0x7f7cc40fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:27.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.974+0000 7f7cca000700 1 -- 172.21.15.84:0/3069968517 shutdown_connections 2024-04-28T12:50:27.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:27.975+0000 7f7cca000700 1 -- 172.21.15.84:0/3069968517 wait complete. 2024-04-28T12:50:28.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.062+0000 7f237d200700 1 -- 172.21.15.84:0/120104106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f237806b130 msgr2=0x7f237806b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:28.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.062+0000 7f237d200700 1 --2- 172.21.15.84:0/120104106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f237806b130 0x7f237806b510 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2368009990 tx=0x7f2368009ca0 comp rx=0 tx=0).stop 2024-04-28T12:50:28.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.062+0000 7f237d200700 1 -- 172.21.15.84:0/120104106 shutdown_connections 2024-04-28T12:50:28.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.062+0000 7f237d200700 1 --2- 172.21.15.84:0/120104106 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f237810fed0 0x7f23781122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:28.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.062+0000 7f237d200700 1 --2- 172.21.15.84:0/120104106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f237806b130 0x7f237806b510 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:28.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.062+0000 7f237d200700 1 --2- 172.21.15.84:0/120104106 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f237806ba50 0x7f237810f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:28.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.062+0000 7f237d200700 1 -- 172.21.15.84:0/120104106 >> 172.21.15.84:0/120104106 conn(0x7f23780792d0 msgr2=0x7f23780796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:28.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.062+0000 7f237d200700 1 -- 172.21.15.84:0/120104106 shutdown_connections 2024-04-28T12:50:28.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.062+0000 7f237d200700 1 -- 172.21.15.84:0/120104106 wait complete. 2024-04-28T12:50:28.060 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:27 smithi084 ceph-mon[28081]: pgmap v381: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:28.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.064+0000 7f237d200700 1 Processor -- start 2024-04-28T12:50:28.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.064+0000 7f237d200700 1 -- start start 2024-04-28T12:50:28.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.064+0000 7f237d200700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f237806b130 0x7f237819be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:28.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.064+0000 7f237d200700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f237806ba50 0x7f237819c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:28.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.064+0000 7f237d200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f237810fed0 0x7f23781a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:28.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.064+0000 7f237d200700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f2378115830 con 0x7f237806b130 2024-04-28T12:50:28.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.064+0000 7f237d200700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f23781156b0 con 0x7f237810fed0 2024-04-28T12:50:28.061 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.064+0000 7f237d200700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f23781159b0 con 0x7f237806ba50 2024-04-28T12:50:28.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.065+0000 7f2375e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f237806b130 0x7f237819be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:28.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.065+0000 7f2375e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f237806b130 0x7f237819be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:51048/0 (socket says 172.21.15.84:51048) 2024-04-28T12:50:28.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.065+0000 7f2375e00700 1 -- 172.21.15.84:0/2051858568 learned_addr learned my addr 172.21.15.84:0/2051858568 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:28.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.065+0000 7f2375400700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f237806ba50 0x7f237819c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:28.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.065+0000 7f2376800700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f237810fed0 0x7f23781a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.065+0000 7f2375400700 1 -- 172.21.15.84:0/2051858568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f237810fed0 msgr2=0x7f23781a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.065+0000 7f2375400700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f237810fed0 0x7f23781a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.065+0000 7f2375400700 1 -- 172.21.15.84:0/2051858568 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f237806b130 msgr2=0x7f237819be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.065+0000 7f2375400700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f237806b130 0x7f237819be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.065+0000 7f2375400700 1 -- 172.21.15.84:0/2051858568 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23781a0e10 con 0x7f237806ba50 2024-04-28T12:50:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.065+0000 7f2375e00700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f237806b130 0x7f237819be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.066+0000 7f2375400700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f237806ba50 0x7f237819c340 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f236400e9f0 tx=0x7f236400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.066+0000 7f236ea00700 1 -- 172.21.15.84:0/2051858568 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236400c9c0 con 0x7f237806ba50 2024-04-28T12:50:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.066+0000 7f236ea00700 1 -- 172.21.15.84:0/2051858568 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f236400cb60 con 0x7f237806ba50 2024-04-28T12:50:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.066+0000 7f236ea00700 1 -- 172.21.15.84:0/2051858568 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364018680 con 0x7f237806ba50 2024-04-28T12:50:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.066+0000 7f237d200700 1 -- 172.21.15.84:0/2051858568 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23781a1100 con 0x7f237806ba50 2024-04-28T12:50:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.066+0000 7f237d200700 1 -- 172.21.15.84:0/2051858568 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23781a8980 con 0x7f237806ba50 2024-04-28T12:50:28.064 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.068+0000 7f237d200700 1 -- 172.21.15.84:0/2051858568 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f237806a110 con 0x7f237806ba50 2024-04-28T12:50:28.066 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.070+0000 7f236ea00700 1 -- 172.21.15.84:0/2051858568 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2364010430 con 0x7f237806ba50 2024-04-28T12:50:28.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.071+0000 7f236ea00700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f235c0a3cd0 0x7f235c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:28.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.071+0000 7f236ea00700 1 -- 172.21.15.84:0/2051858568 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f23640d0aa0 con 0x7f237806ba50 2024-04-28T12:50:28.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.072+0000 7f2375e00700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f235c0a3cd0 0x7f235c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:28.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.072+0000 7f2375e00700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f235c0a3cd0 0x7f235c0a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2368006010 tx=0x7f236800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:28.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.076+0000 7f236ea00700 1 -- 172.21.15.84:0/2051858568 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f236409cf60 con 0x7f237806ba50 2024-04-28T12:50:28.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:27 smithi179 ceph-mon[63029]: pgmap v381: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:28.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:27 smithi161 ceph-mon[62387]: pgmap v381: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:28.320 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.324+0000 7f237d200700 1 -- 172.21.15.84:0/2051858568 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f237802e490 con 0x7f237806ba50 2024-04-28T12:50:28.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.325+0000 7f236ea00700 1 -- 172.21.15.84:0/2051858568 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f236409c860 con 0x7f237806ba50 2024-04-28T12:50:28.321 INFO:teuthology.orchestra.run.smithi084.stdout:287762808880 2024-04-28T12:50:28.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.328+0000 7f237d200700 1 -- 172.21.15.84:0/2051858568 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f235c0a3cd0 msgr2=0x7f235c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:28.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.328+0000 7f237d200700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f235c0a3cd0 0x7f235c0a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2368006010 tx=0x7f236800b280 comp rx=0 tx=0).stop 2024-04-28T12:50:28.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.328+0000 7f237d200700 1 -- 172.21.15.84:0/2051858568 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f237806ba50 msgr2=0x7f237819c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:28.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.328+0000 7f237d200700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f237806ba50 0x7f237819c340 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f236400e9f0 tx=0x7f236400ed00 comp rx=0 tx=0).stop 2024-04-28T12:50:28.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.328+0000 7f237d200700 1 -- 172.21.15.84:0/2051858568 shutdown_connections 2024-04-28T12:50:28.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.328+0000 7f237d200700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f235c0a3cd0 0x7f235c0a6190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:28.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.328+0000 7f237d200700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f237810fed0 0x7f23781a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:28.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.328+0000 7f237d200700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f237806b130 0x7f237819be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:28.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.328+0000 7f237d200700 1 --2- 172.21.15.84:0/2051858568 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f237806ba50 0x7f237819c340 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:28.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.328+0000 7f237d200700 1 -- 172.21.15.84:0/2051858568 >> 172.21.15.84:0/2051858568 conn(0x7f23780792d0 msgr2=0x7f2378111610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:28.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.329+0000 7f237d200700 1 -- 172.21.15.84:0/2051858568 shutdown_connections 2024-04-28T12:50:28.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:28.329+0000 7f237d200700 1 -- 172.21.15.84:0/2051858568 wait complete. 2024-04-28T12:50:29.094 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:28 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3069968517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-28T12:50:29.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:28 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2051858568' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-28T12:50:29.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:28 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3069968517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-28T12:50:29.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:28 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2051858568' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-28T12:50:29.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:28 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3069968517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-28T12:50:29.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:28 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2051858568' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-28T12:50:29.996 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:29 smithi084 ceph-mon[28081]: pgmap v382: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:29.996 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:29 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3461003657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-28T12:50:29.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-28T12:50:29.999 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411412 got 77309411418 for osd.2 2024-04-28T12:50:29.999 DEBUG:teuthology.parallel:result is None 2024-04-28T12:50:30.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:29 smithi179 ceph-mon[63029]: pgmap v382: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:30.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:29 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3461003657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-28T12:50:30.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-28T12:50:30.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:29 smithi161 ceph-mon[62387]: pgmap v382: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:30.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:29 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3461003657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-28T12:50:30.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-28T12:50:30.409 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574935 got 55834574941 for osd.1 2024-04-28T12:50:30.409 DEBUG:teuthology.parallel:result is None 2024-04-28T12:50:30.516 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738457 got 34359738466 for osd.0 2024-04-28T12:50:30.516 DEBUG:teuthology.parallel:result is None 2024-04-28T12:50:30.648 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070511 got 253403070516 for osd.8 2024-04-28T12:50:30.648 DEBUG:teuthology.parallel:result is None 2024-04-28T12:50:30.758 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.481+0000 7f6d72600700 1 -- 172.21.15.84:0/3688800646 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6c106000 msgr2=0x7f6d6c112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:30.758 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.481+0000 7f6d72600700 1 --2- 172.21.15.84:0/3688800646 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6c106000 0x7f6d6c112510 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6d6400b0a0 tx=0x7f6d6400b3b0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.758 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.481+0000 7f6d72600700 1 -- 172.21.15.84:0/3688800646 shutdown_connections 2024-04-28T12:50:30.758 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.481+0000 7f6d72600700 1 --2- 172.21.15.84:0/3688800646 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6c106000 0x7f6d6c112510 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.758 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.481+0000 7f6d72600700 1 --2- 172.21.15.84:0/3688800646 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d6c10b640 0x7f6d6c10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.481+0000 7f6d72600700 1 --2- 172.21.15.84:0/3688800646 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d6c105640 0x7f6d6c105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.481+0000 7f6d72600700 1 -- 172.21.15.84:0/3688800646 >> 172.21.15.84:0/3688800646 conn(0x7f6d6c101130 msgr2=0x7f6d6c103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.481+0000 7f6d72600700 1 -- 172.21.15.84:0/3688800646 shutdown_connections 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.481+0000 7f6d72600700 1 -- 172.21.15.84:0/3688800646 wait complete. 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.482+0000 7f6d72600700 1 Processor -- start 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.482+0000 7f6d72600700 1 -- start start 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d72600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d6c105640 0x7f6d6c1a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d72600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d6c106000 0x7f6d6c1a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d72600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6c10b640 0x7f6d6c19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d72600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c1158d0 con 0x7f6d6c10b640 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d72600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c115750 con 0x7f6d6c106000 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d72600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c115a50 con 0x7f6d6c105640 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d6be00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6c10b640 0x7f6d6c19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d6b400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d6c105640 0x7f6d6c1a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d6aa00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d6c106000 0x7f6d6c1a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:30.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d6b400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d6c105640 0x7f6d6c1a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:41802/0 (socket says 172.21.15.84:41802) 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d6b400700 1 -- 172.21.15.84:0/3461003657 learned_addr learned my addr 172.21.15.84:0/3461003657 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d6be00700 1 -- 172.21.15.84:0/3461003657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d6c105640 msgr2=0x7f6d6c1a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d6be00700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d6c105640 0x7f6d6c1a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d6be00700 1 -- 172.21.15.84:0/3461003657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d6c106000 msgr2=0x7f6d6c1a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.483+0000 7f6d6be00700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d6c106000 0x7f6d6c1a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.484+0000 7f6d6be00700 1 -- 172.21.15.84:0/3461003657 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d6c19c230 con 0x7f6d6c10b640 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.484+0000 7f6d6aa00700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d6c106000 0x7f6d6c1a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.484+0000 7f6d6be00700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6c10b640 0x7f6d6c19bad0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6d6400b070 tx=0x7f6d64004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.484+0000 7f6d63e00700 1 -- 172.21.15.84:0/3461003657 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d64021070 con 0x7f6d6c10b640 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.484+0000 7f6d63e00700 1 -- 172.21.15.84:0/3461003657 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d64007e30 con 0x7f6d6c10b640 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.484+0000 7f6d63e00700 1 -- 172.21.15.84:0/3461003657 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d640045f0 con 0x7f6d6c10b640 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.484+0000 7f6d72600700 1 -- 172.21.15.84:0/3461003657 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d6c19c4c0 con 0x7f6d6c10b640 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.484+0000 7f6d72600700 1 -- 172.21.15.84:0/3461003657 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d6c1a87e0 con 0x7f6d6c10b640 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.486+0000 7f6d72600700 1 -- 172.21.15.84:0/3461003657 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d6c106d80 con 0x7f6d6c10b640 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.488+0000 7f6d63e00700 1 -- 172.21.15.84:0/3461003657 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6d64007980 con 0x7f6d6c10b640 2024-04-28T12:50:30.760 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.489+0000 7f6d63e00700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6d500a3d50 0x7f6d500a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:30.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.489+0000 7f6d63e00700 1 -- 172.21.15.84:0/3461003657 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f6d640d16a0 con 0x7f6d6c10b640 2024-04-28T12:50:30.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.489+0000 7f6d6b400700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6d500a3d50 0x7f6d500a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:30.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.490+0000 7f6d6b400700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6d500a3d50 0x7f6d500a6210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009ce0 tx=0x7f6d5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:30.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.494+0000 7f6d63e00700 1 -- 172.21.15.84:0/3461003657 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6d6409dbf0 con 0x7f6d6c10b640 2024-04-28T12:50:30.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.724+0000 7f6d72600700 1 -- 172.21.15.84:0/3461003657 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f6d6c02e490 con 0x7f6d6c10b640 2024-04-28T12:50:30.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.725+0000 7f6d63e00700 1 -- 172.21.15.84:0/3461003657 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6d6409d4f0 con 0x7f6d6c10b640 2024-04-28T12:50:30.761 INFO:teuthology.orchestra.run.smithi084.stdout:107374182483 2024-04-28T12:50:30.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.728+0000 7f6d72600700 1 -- 172.21.15.84:0/3461003657 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6d500a3d50 msgr2=0x7f6d500a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:30.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.728+0000 7f6d72600700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6d500a3d50 0x7f6d500a6210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009ce0 tx=0x7f6d5c009260 comp rx=0 tx=0).stop 2024-04-28T12:50:30.761 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.728+0000 7f6d72600700 1 -- 172.21.15.84:0/3461003657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6c10b640 msgr2=0x7f6d6c19bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:30.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.728+0000 7f6d72600700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6c10b640 0x7f6d6c19bad0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6d6400b070 tx=0x7f6d64004c80 comp rx=0 tx=0).stop 2024-04-28T12:50:30.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.728+0000 7f6d72600700 1 -- 172.21.15.84:0/3461003657 shutdown_connections 2024-04-28T12:50:30.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.728+0000 7f6d72600700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d6c10b640 0x7f6d6c19bad0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.728+0000 7f6d72600700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6d6c105640 0x7f6d6c1a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.728+0000 7f6d72600700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6d500a3d50 0x7f6d500a6210 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.728+0000 7f6d72600700 1 --2- 172.21.15.84:0/3461003657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d6c106000 0x7f6d6c1a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:30.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.728+0000 7f6d72600700 1 -- 172.21.15.84:0/3461003657 >> 172.21.15.84:0/3461003657 conn(0x7f6d6c101130 msgr2=0x7f6d6c1016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:30.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.728+0000 7f6d72600700 1 -- 172.21.15.84:0/3461003657 shutdown_connections 2024-04-28T12:50:30.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:29.729+0000 7f6d72600700 1 -- 172.21.15.84:0/3461003657 wait complete. 2024-04-28T12:50:30.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:30 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-28T12:50:30.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:30 smithi084 ceph-mon[28081]: osdmap e93: 12 total, 12 up, 12 in 2024-04-28T12:50:31.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:30 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-28T12:50:31.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:30 smithi179 ceph-mon[63029]: osdmap e93: 12 total, 12 up, 12 in 2024-04-28T12:50:31.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:30 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-28T12:50:31.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:30 smithi161 ceph-mon[62387]: osdmap e93: 12 total, 12 up, 12 in 2024-04-28T12:50:31.541 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182476 got 107374182483 for osd.3 2024-04-28T12:50:31.542 DEBUG:teuthology.parallel:result is None 2024-04-28T12:50:31.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.803+0000 7fcc43600700 1 -- 172.21.15.84:0/3494592694 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc3c06bad0 msgr2=0x7fcc3c1102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:31.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.803+0000 7fcc43600700 1 --2- 172.21.15.84:0/3494592694 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc3c06bad0 0x7fcc3c1102e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009f90 tx=0x7fcc3400b330 comp rx=0 tx=0).stop 2024-04-28T12:50:31.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.804+0000 7fcc43600700 1 -- 172.21.15.84:0/3494592694 shutdown_connections 2024-04-28T12:50:31.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.804+0000 7fcc43600700 1 --2- 172.21.15.84:0/3494592694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc3c1094a0 0x7fcc3c109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.804+0000 7fcc43600700 1 --2- 172.21.15.84:0/3494592694 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc3c06bad0 0x7fcc3c1102e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.804+0000 7fcc43600700 1 --2- 172.21.15.84:0/3494592694 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc3c06b130 0x7fcc3c06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.804+0000 7fcc43600700 1 -- 172.21.15.84:0/3494592694 >> 172.21.15.84:0/3494592694 conn(0x7fcc3c0792d0 msgr2=0x7fcc3c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:31.800 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.804+0000 7fcc43600700 1 -- 172.21.15.84:0/3494592694 shutdown_connections 2024-04-28T12:50:31.801 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.804+0000 7fcc43600700 1 -- 172.21.15.84:0/3494592694 wait complete. 2024-04-28T12:50:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.805+0000 7fcc43600700 1 Processor -- start 2024-04-28T12:50:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.805+0000 7fcc43600700 1 -- start start 2024-04-28T12:50:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.805+0000 7fcc43600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc3c06b130 0x7fcc3c1023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.805+0000 7fcc43600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc3c06bad0 0x7fcc3c102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.805+0000 7fcc43600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc3c1094a0 0x7fcc3c102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.805+0000 7fcc43600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c1135f0 con 0x7fcc3c06b130 2024-04-28T12:50:31.802 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.805+0000 7fcc43600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c113470 con 0x7fcc3c1094a0 2024-04-28T12:50:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.805+0000 7fcc43600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3c113770 con 0x7fcc3c06bad0 2024-04-28T12:50:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.806+0000 7fcc40c00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc3c06b130 0x7fcc3c1023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.806+0000 7fcc40c00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc3c06b130 0x7fcc3c1023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:57968/0 (socket says 172.21.15.84:57968) 2024-04-28T12:50:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.806+0000 7fcc40c00700 1 -- 172.21.15.84:0/1294267821 learned_addr learned my addr 172.21.15.84:0/1294267821 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.806+0000 7fcc41600700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc3c1094a0 0x7fcc3c102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.806+0000 7fcc3be00700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc3c06bad0 0x7fcc3c102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.806+0000 7fcc40c00700 1 -- 172.21.15.84:0/1294267821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc3c06bad0 msgr2=0x7fcc3c102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.806+0000 7fcc40c00700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc3c06bad0 0x7fcc3c102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.806+0000 7fcc40c00700 1 -- 172.21.15.84:0/1294267821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc3c1094a0 msgr2=0x7fcc3c102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.806+0000 7fcc40c00700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc3c1094a0 0x7fcc3c102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.806+0000 7fcc40c00700 1 -- 172.21.15.84:0/1294267821 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3c1a80e0 con 0x7fcc3c06b130 2024-04-28T12:50:31.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.806+0000 7fcc41600700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc3c1094a0 0x7fcc3c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:31.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.807+0000 7fcc40c00700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc3c06b130 0x7fcc3c1023f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000e9a0 tx=0x7fcc3000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:31.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.807+0000 7fcc39600700 1 -- 172.21.15.84:0/1294267821 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3000ca10 con 0x7fcc3c06b130 2024-04-28T12:50:31.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.807+0000 7fcc43600700 1 -- 172.21.15.84:0/1294267821 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3c1a83d0 con 0x7fcc3c06b130 2024-04-28T12:50:31.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.807+0000 7fcc39600700 1 -- 172.21.15.84:0/1294267821 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc3000cbb0 con 0x7fcc3c06b130 2024-04-28T12:50:31.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.807+0000 7fcc39600700 1 -- 172.21.15.84:0/1294267821 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30018680 con 0x7fcc3c06b130 2024-04-28T12:50:31.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.807+0000 7fcc43600700 1 -- 172.21.15.84:0/1294267821 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc3c1a8910 con 0x7fcc3c06b130 2024-04-28T12:50:31.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.808+0000 7fcc43600700 1 -- 172.21.15.84:0/1294267821 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc3c06a110 con 0x7fcc3c06b130 2024-04-28T12:50:31.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.810+0000 7fcc39600700 1 -- 172.21.15.84:0/1294267821 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcc30010430 con 0x7fcc3c06b130 2024-04-28T12:50:31.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.811+0000 7fcc39600700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcc240a3cd0 0x7fcc240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:31.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.812+0000 7fcc39600700 1 -- 172.21.15.84:0/1294267821 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fcc300d09e0 con 0x7fcc3c06b130 2024-04-28T12:50:31.808 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.812+0000 7fcc3be00700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcc240a3cd0 0x7fcc240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:31.808 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.813+0000 7fcc3be00700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcc240a3cd0 0x7fcc240a6190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009750 tx=0x7fcc28009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:31.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:31.816+0000 7fcc39600700 1 -- 172.21.15.84:0/1294267821 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcc3009cee0 con 0x7fcc3c06b130 2024-04-28T12:50:32.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.059+0000 7fcc43600700 1 -- 172.21.15.84:0/1294267821 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fcc3c02e490 con 0x7fcc3c06b130 2024-04-28T12:50:32.056 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:31 smithi084 ceph-mon[28081]: osdmap e94: 12 total, 12 up, 12 in 2024-04-28T12:50:32.056 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:31 smithi084 ceph-mon[28081]: pgmap v385: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:32.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.060+0000 7fcc39600700 1 -- 172.21.15.84:0/1294267821 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fcc3009c7e0 con 0x7fcc3c06b130 2024-04-28T12:50:32.056 INFO:teuthology.orchestra.run.smithi084.stdout:141733920846 2024-04-28T12:50:32.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.063+0000 7fcc43600700 1 -- 172.21.15.84:0/1294267821 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcc240a3cd0 msgr2=0x7fcc240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.063+0000 7fcc43600700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcc240a3cd0 0x7fcc240a6190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009750 tx=0x7fcc28009310 comp rx=0 tx=0).stop 2024-04-28T12:50:32.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.063+0000 7fcc43600700 1 -- 172.21.15.84:0/1294267821 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc3c06b130 msgr2=0x7fcc3c1023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:32.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.063+0000 7fcc43600700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc3c06b130 0x7fcc3c1023f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000e9a0 tx=0x7fcc3000ecb0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.063+0000 7fcc43600700 1 -- 172.21.15.84:0/1294267821 shutdown_connections 2024-04-28T12:50:32.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.063+0000 7fcc43600700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fcc240a3cd0 0x7fcc240a6190 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.063+0000 7fcc43600700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc3c1094a0 0x7fcc3c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.063+0000 7fcc43600700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc3c06b130 0x7fcc3c1023f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.063+0000 7fcc43600700 1 --2- 172.21.15.84:0/1294267821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc3c06bad0 0x7fcc3c102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:32.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.063+0000 7fcc43600700 1 -- 172.21.15.84:0/1294267821 >> 172.21.15.84:0/1294267821 conn(0x7fcc3c0792d0 msgr2=0x7fcc3c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:32.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.063+0000 7fcc43600700 1 -- 172.21.15.84:0/1294267821 shutdown_connections 2024-04-28T12:50:32.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:32.064+0000 7fcc43600700 1 -- 172.21.15.84:0/1294267821 wait complete. 2024-04-28T12:50:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:31 smithi179 ceph-mon[63029]: osdmap e94: 12 total, 12 up, 12 in 2024-04-28T12:50:32.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:31 smithi179 ceph-mon[63029]: pgmap v385: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:32.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:31 smithi161 ceph-mon[62387]: osdmap e94: 12 total, 12 up, 12 in 2024-04-28T12:50:32.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:31 smithi161 ceph-mon[62387]: pgmap v385: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:32.432 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808873 got 287762808880 for osd.9 2024-04-28T12:50:32.432 DEBUG:teuthology.parallel:result is None 2024-04-28T12:50:32.961 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383710 got 343597383715 for osd.11 2024-04-28T12:50:32.961 DEBUG:teuthology.parallel:result is None 2024-04-28T12:50:33.144 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:32 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1294267821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-28T12:50:33.149 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:32 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1294267821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-28T12:50:33.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:32 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1294267821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-28T12:50:33.729 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920840 got 141733920846 for osd.4 2024-04-28T12:50:33.729 DEBUG:teuthology.parallel:result is None 2024-04-28T12:50:33.730 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-28T12:50:33.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-28T12:50:33.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:33 smithi084 ceph-mon[28081]: pgmap v386: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-28T12:50:34.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:33 smithi179 ceph-mon[63029]: pgmap v386: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-28T12:50:34.185 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:34.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:33 smithi161 ceph-mon[62387]: pgmap v386: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 127 B/s wr, 4 op/s 2024-04-28T12:50:35.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:35 smithi161 ceph-mon[62387]: pgmap v387: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 7 op/s 2024-04-28T12:50:36.029 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:35 smithi084 ceph-mon[28081]: pgmap v387: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 7 op/s 2024-04-28T12:50:36.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:35 smithi179 ceph-mon[63029]: pgmap v387: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 7 op/s 2024-04-28T12:50:36.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.594+0000 7f43ad400700 1 -- 172.21.15.84:0/4031960972 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f43a8105f10 msgr2=0x7f43a81123d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:36.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.594+0000 7f43ad400700 1 --2- 172.21.15.84:0/4031960972 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f43a8105f10 0x7f43a81123d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f43a000b0d0 tx=0x7f43a000b3e0 comp rx=0 tx=0).stop 2024-04-28T12:50:36.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.595+0000 7f43ad400700 1 -- 172.21.15.84:0/4031960972 shutdown_connections 2024-04-28T12:50:36.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.595+0000 7f43ad400700 1 --2- 172.21.15.84:0/4031960972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43a810b600 0x7f43a810b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:36.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.595+0000 7f43ad400700 1 --2- 172.21.15.84:0/4031960972 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f43a8105f10 0x7f43a81123d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:36.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.595+0000 7f43ad400700 1 --2- 172.21.15.84:0/4031960972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f43a8105550 0x7f43a81059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:36.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.595+0000 7f43ad400700 1 -- 172.21.15.84:0/4031960972 >> 172.21.15.84:0/4031960972 conn(0x7f43a8101060 msgr2=0x7f43a8103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:36.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.595+0000 7f43ad400700 1 -- 172.21.15.84:0/4031960972 shutdown_connections 2024-04-28T12:50:36.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.595+0000 7f43ad400700 1 -- 172.21.15.84:0/4031960972 wait complete. 2024-04-28T12:50:36.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.596+0000 7f43ad400700 1 Processor -- start 2024-04-28T12:50:36.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.596+0000 7f43ad400700 1 -- start start 2024-04-28T12:50:36.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.597+0000 7f43ad400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f43a8105550 0x7f43a81a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:36.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.597+0000 7f43ad400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43a8105f10 0x7f43a81a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:36.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.597+0000 7f43ad400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f43a810b600 0x7f43a819b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:36.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.597+0000 7f43ad400700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f43a8115770 con 0x7f43a8105550 2024-04-28T12:50:36.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.597+0000 7f43ad400700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f43a81155f0 con 0x7f43a8105f10 2024-04-28T12:50:36.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.597+0000 7f43ad400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f43a81158f0 con 0x7f43a810b600 2024-04-28T12:50:36.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.597+0000 7f43a7e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f43a8105550 0x7f43a81a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:36.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.597+0000 7f43a7e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f43a8105550 0x7f43a81a1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:57974/0 (socket says 172.21.15.84:57974) 2024-04-28T12:50:36.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.597+0000 7f43a7e00700 1 -- 172.21.15.84:0/2129830289 learned_addr learned my addr 172.21.15.84:0/2129830289 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:36.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.597+0000 7f43aca00700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f43a810b600 0x7f43a819b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:36.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.597+0000 7f43a7400700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43a8105f10 0x7f43a81a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:36.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.598+0000 7f43aca00700 1 -- 172.21.15.84:0/2129830289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43a8105f10 msgr2=0x7f43a81a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:36.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.598+0000 7f43aca00700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43a8105f10 0x7f43a81a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:36.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.598+0000 7f43aca00700 1 -- 172.21.15.84:0/2129830289 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f43a8105550 msgr2=0x7f43a81a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:36.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.598+0000 7f43aca00700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f43a8105550 0x7f43a81a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:36.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.598+0000 7f43aca00700 1 -- 172.21.15.84:0/2129830289 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a819c090 con 0x7f43a810b600 2024-04-28T12:50:36.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.598+0000 7f43a7e00700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f43a8105550 0x7f43a81a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:36.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.598+0000 7f43aca00700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f43a810b600 0x7f43a819b930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f43a000b770 tx=0x7f43a0007980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:36.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.598+0000 7f43a4c00700 1 -- 172.21.15.84:0/2129830289 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a001f070 con 0x7f43a810b600 2024-04-28T12:50:36.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.598+0000 7f43a4c00700 1 -- 172.21.15.84:0/2129830289 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43a0007dc0 con 0x7f43a810b600 2024-04-28T12:50:36.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.598+0000 7f43ad400700 1 -- 172.21.15.84:0/2129830289 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a819c320 con 0x7f43a810b600 2024-04-28T12:50:36.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.598+0000 7f43a4c00700 1 -- 172.21.15.84:0/2129830289 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a001b430 con 0x7f43a810b600 2024-04-28T12:50:36.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.599+0000 7f43ad400700 1 -- 172.21.15.84:0/2129830289 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a81a8640 con 0x7f43a810b600 2024-04-28T12:50:36.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.600+0000 7f43ad400700 1 -- 172.21.15.84:0/2129830289 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43a8106c70 con 0x7f43a810b600 2024-04-28T12:50:36.598 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.602+0000 7f43a4c00700 1 -- 172.21.15.84:0/2129830289 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f43a0022070 con 0x7f43a810b600 2024-04-28T12:50:36.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.603+0000 7f43a4c00700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f438c0a3cd0 0x7f438c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:36.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.603+0000 7f43a4c00700 1 -- 172.21.15.84:0/2129830289 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f43a000f020 con 0x7f43a810b600 2024-04-28T12:50:36.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.603+0000 7f43a7e00700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f438c0a3cd0 0x7f438c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:36.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.604+0000 7f43a7e00700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f438c0a3cd0 0x7f438c0a6190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f43a819ca70 tx=0x7f4398009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:36.604 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.608+0000 7f43a4c00700 1 -- 172.21.15.84:0/2129830289 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f43a0029020 con 0x7f43a810b600 2024-04-28T12:50:36.826 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.830+0000 7f43ad400700 1 -- 172.21.15.84:0/2129830289 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f43a819ce60 con 0x7f438c0a3cd0 2024-04-28T12:50:36.839 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.843+0000 7f43a4c00700 1 -- 172.21.15.84:0/2129830289 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340700 (secure 0 0 0) 0x7f43a819ce60 con 0x7f438c0a3cd0 2024-04-28T12:50:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:50:36.848 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.849+0000 7f43ad400700 1 -- 172.21.15.84:0/2129830289 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f438c0a3cd0 msgr2=0x7f438c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:36.849 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.849+0000 7f43ad400700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f438c0a3cd0 0x7f438c0a6190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f43a819ca70 tx=0x7f4398009260 comp rx=0 tx=0).stop 2024-04-28T12:50:36.849 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.849+0000 7f43ad400700 1 -- 172.21.15.84:0/2129830289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f43a810b600 msgr2=0x7f43a819b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:36.849 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.849+0000 7f43ad400700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f43a810b600 0x7f43a819b930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f43a000b770 tx=0x7f43a0007980 comp rx=0 tx=0).stop 2024-04-28T12:50:36.849 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.849+0000 7f43ad400700 1 -- 172.21.15.84:0/2129830289 shutdown_connections 2024-04-28T12:50:36.849 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.849+0000 7f43ad400700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f43a810b600 0x7f43a819b930 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:36.849 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.849+0000 7f43ad400700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f43a8105550 0x7f43a81a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:36.849 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.849+0000 7f43ad400700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f438c0a3cd0 0x7f438c0a6190 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:36.849 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.849+0000 7f43ad400700 1 --2- 172.21.15.84:0/2129830289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43a8105f10 0x7f43a81a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:36.849 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.849+0000 7f43ad400700 1 -- 172.21.15.84:0/2129830289 >> 172.21.15.84:0/2129830289 conn(0x7f43a8101060 msgr2=0x7f43a81026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:36.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.849+0000 7f43ad400700 1 -- 172.21.15.84:0/2129830289 shutdown_connections 2024-04-28T12:50:36.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:36.850+0000 7f43ad400700 1 -- 172.21.15.84:0/2129830289 wait complete. 2024-04-28T12:50:36.850 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-28T12:50:37.715 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-04-28T12:50:35.653478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649820,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":672,"num_read_kb":445,"num_write":347,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":9,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365924,"kb_used_data":12548,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575900,"statfs":{"total":1151940427776,"available":1151565721600,"internally_reserved":0,"allocated":12849152,"data_stored":8238904,"data_compressed":421859,"data_compressed_allocated":569344,"data_compressed_original":2637824,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002781"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118023+0000","last_change":"2024-04-28T12:48:34.569210+0000","last_active":"2024-04-28T12:50:31.118023+0000","last_peered":"2024-04-28T12:50:31.118023+0000","last_clean":"2024-04-28T12:50:31.118023+0000","last_became_active":"2024-04-28T12:48:34.568912+0000","last_became_peered":"2024-04-28T12:48:34.568912+0000","last_unstale":"2024-04-28T12:50:31.118023+0000","last_undegraded":"2024-04-28T12:50:31.118023+0000","last_fullsized":"2024-04-28T12:50:31.118023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:48:26.513370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.120476+0000","last_change":"2024-04-28T12:48:34.577664+0000","last_active":"2024-04-28T12:50:31.120476+0000","last_peered":"2024-04-28T12:50:31.120476+0000","last_clean":"2024-04-28T12:50:31.120476+0000","last_became_active":"2024-04-28T12:48:34.576679+0000","last_became_peered":"2024-04-28T12:48:34.576679+0000","last_unstale":"2024-04-28T12:50:31.120476+0000","last_undegraded":"2024-04-28T12:50:31.120476+0000","last_fullsized":"2024-04-28T12:50:31.120476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:26:44.807081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.119777+0000","last_change":"2024-04-28T12:48:34.573666+0000","last_active":"2024-04-28T12:50:31.119777+0000","last_peered":"2024-04-28T12:50:31.119777+0000","last_clean":"2024-04-28T12:50:31.119777+0000","last_became_active":"2024-04-28T12:48:34.573111+0000","last_became_peered":"2024-04-28T12:48:34.573111+0000","last_unstale":"2024-04-28T12:50:31.119777+0000","last_undegraded":"2024-04-28T12:50:31.119777+0000","last_fullsized":"2024-04-28T12:50:31.119777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:41:13.586589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.909372+0000","last_change":"2024-04-28T12:48:34.543140+0000","last_active":"2024-04-28T12:50:30.909372+0000","last_peered":"2024-04-28T12:50:30.909372+0000","last_clean":"2024-04-28T12:50:30.909372+0000","last_became_active":"2024-04-28T12:48:34.542803+0000","last_became_peered":"2024-04-28T12:48:34.542803+0000","last_unstale":"2024-04-28T12:50:30.909372+0000","last_undegraded":"2024-04-28T12:50:30.909372+0000","last_fullsized":"2024-04-28T12:50:30.909372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:46:47.313190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998318+0000","last_change":"2024-04-28T12:48:34.559988+0000","last_active":"2024-04-28T12:50:30.998318+0000","last_peered":"2024-04-28T12:50:30.998318+0000","last_clean":"2024-04-28T12:50:30.998318+0000","last_became_active":"2024-04-28T12:48:34.559657+0000","last_became_peered":"2024-04-28T12:48:34.559657+0000","last_unstale":"2024-04-28T12:50:30.998318+0000","last_undegraded":"2024-04-28T12:50:30.998318+0000","last_fullsized":"2024-04-28T12:50:30.998318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:13:27.610212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.855834+0000","last_change":"2024-04-28T12:48:34.553539+0000","last_active":"2024-04-28T12:50:30.855834+0000","last_peered":"2024-04-28T12:50:30.855834+0000","last_clean":"2024-04-28T12:50:30.855834+0000","last_became_active":"2024-04-28T12:48:34.553017+0000","last_became_peered":"2024-04-28T12:48:34.553017+0000","last_unstale":"2024-04-28T12:50:30.855834+0000","last_undegraded":"2024-04-28T12:50:30.855834+0000","last_fullsized":"2024-04-28T12:50:30.855834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:31:14.895894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117554+0000","last_change":"2024-04-28T12:48:34.553305+0000","last_active":"2024-04-28T12:50:31.117554+0000","last_peered":"2024-04-28T12:50:31.117554+0000","last_clean":"2024-04-28T12:50:31.117554+0000","last_became_active":"2024-04-28T12:48:34.552950+0000","last_became_peered":"2024-04-28T12:48:34.552950+0000","last_unstale":"2024-04-28T12:50:31.117554+0000","last_undegraded":"2024-04-28T12:50:31.117554+0000","last_fullsized":"2024-04-28T12:50:31.117554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:01:43.903986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.859871+0000","last_change":"2024-04-28T12:48:34.558899+0000","last_active":"2024-04-28T12:50:30.859871+0000","last_peered":"2024-04-28T12:50:30.859871+0000","last_clean":"2024-04-28T12:50:30.859871+0000","last_became_active":"2024-04-28T12:48:34.558507+0000","last_became_peered":"2024-04-28T12:48:34.558507+0000","last_unstale":"2024-04-28T12:50:30.859871+0000","last_undegraded":"2024-04-28T12:50:30.859871+0000","last_fullsized":"2024-04-28T12:50:30.859871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:46:28.059231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.272190+0000","last_change":"2024-04-28T12:48:34.545488+0000","last_active":"2024-04-28T12:50:30.272190+0000","last_peered":"2024-04-28T12:50:30.272190+0000","last_clean":"2024-04-28T12:50:30.272190+0000","last_became_active":"2024-04-28T12:48:34.545168+0000","last_became_peered":"2024-04-28T12:48:34.545168+0000","last_unstale":"2024-04-28T12:50:30.272190+0000","last_undegraded":"2024-04-28T12:50:30.272190+0000","last_fullsized":"2024-04-28T12:50:30.272190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:18:02.174760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.856319+0000","last_change":"2024-04-28T12:48:34.544213+0000","last_active":"2024-04-28T12:50:30.856319+0000","last_peered":"2024-04-28T12:50:30.856319+0000","last_clean":"2024-04-28T12:50:30.856319+0000","last_became_active":"2024-04-28T12:48:34.543910+0000","last_became_peered":"2024-04-28T12:48:34.543910+0000","last_unstale":"2024-04-28T12:50:30.856319+0000","last_undegraded":"2024-04-28T12:50:30.856319+0000","last_fullsized":"2024-04-28T12:50:30.856319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:09:52.602695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118151+0000","last_change":"2024-04-28T12:48:34.548121+0000","last_active":"2024-04-28T12:50:31.118151+0000","last_peered":"2024-04-28T12:50:31.118151+0000","last_clean":"2024-04-28T12:50:31.118151+0000","last_became_active":"2024-04-28T12:48:34.547822+0000","last_became_peered":"2024-04-28T12:48:34.547822+0000","last_unstale":"2024-04-28T12:50:31.118151+0000","last_undegraded":"2024-04-28T12:50:31.118151+0000","last_fullsized":"2024-04-28T12:50:31.118151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T12:49:39.838959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.909308+0000","last_change":"2024-04-28T12:48:34.544146+0000","last_active":"2024-04-28T12:50:30.909308+0000","last_peered":"2024-04-28T12:50:30.909308+0000","last_clean":"2024-04-28T12:50:30.909308+0000","last_became_active":"2024-04-28T12:48:34.543805+0000","last_became_peered":"2024-04-28T12:48:34.543805+0000","last_unstale":"2024-04-28T12:50:30.909308+0000","last_undegraded":"2024-04-28T12:50:30.909308+0000","last_fullsized":"2024-04-28T12:50:30.909308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:06:46.871316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.906406+0000","last_change":"2024-04-28T12:48:34.536765+0000","last_active":"2024-04-28T12:50:30.906406+0000","last_peered":"2024-04-28T12:50:30.906406+0000","last_clean":"2024-04-28T12:50:30.906406+0000","last_became_active":"2024-04-28T12:48:34.535398+0000","last_became_peered":"2024-04-28T12:48:34.535398+0000","last_unstale":"2024-04-28T12:50:30.906406+0000","last_undegraded":"2024-04-28T12:50:30.906406+0000","last_fullsized":"2024-04-28T12:50:30.906406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:54:25.100414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.119606+0000","last_change":"2024-04-28T12:48:34.573782+0000","last_active":"2024-04-28T12:50:31.119606+0000","last_peered":"2024-04-28T12:50:31.119606+0000","last_clean":"2024-04-28T12:50:31.119606+0000","last_became_active":"2024-04-28T12:48:34.573361+0000","last_became_peered":"2024-04-28T12:48:34.573361+0000","last_unstale":"2024-04-28T12:50:31.119606+0000","last_undegraded":"2024-04-28T12:50:31.119606+0000","last_fullsized":"2024-04-28T12:50:31.119606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:29:02.746038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.119979+0000","last_change":"2024-04-28T12:48:34.570810+0000","last_active":"2024-04-28T12:50:31.119979+0000","last_peered":"2024-04-28T12:50:31.119979+0000","last_clean":"2024-04-28T12:50:31.119979+0000","last_became_active":"2024-04-28T12:48:34.570337+0000","last_became_peered":"2024-04-28T12:48:34.570337+0000","last_unstale":"2024-04-28T12:50:31.119979+0000","last_undegraded":"2024-04-28T12:50:31.119979+0000","last_fullsized":"2024-04-28T12:50:31.119979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:41:36.028710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.271517+0000","last_change":"2024-04-28T12:48:34.558802+0000","last_active":"2024-04-28T12:50:30.271517+0000","last_peered":"2024-04-28T12:50:30.271517+0000","last_clean":"2024-04-28T12:50:30.271517+0000","last_became_active":"2024-04-28T12:48:34.558425+0000","last_became_peered":"2024-04-28T12:48:34.558425+0000","last_unstale":"2024-04-28T12:50:30.271517+0000","last_undegraded":"2024-04-28T12:50:30.271517+0000","last_fullsized":"2024-04-28T12:50:30.271517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:03:22.681965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117984+0000","last_change":"2024-04-28T12:48:34.569546+0000","last_active":"2024-04-28T12:50:31.117984+0000","last_peered":"2024-04-28T12:50:31.117984+0000","last_clean":"2024-04-28T12:50:31.117984+0000","last_became_active":"2024-04-28T12:48:34.569231+0000","last_became_peered":"2024-04-28T12:48:34.569231+0000","last_unstale":"2024-04-28T12:50:31.117984+0000","last_undegraded":"2024-04-28T12:50:31.117984+0000","last_fullsized":"2024-04-28T12:50:31.117984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:19:24.491204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.869122+0000","last_change":"2024-04-28T12:48:34.555634+0000","last_active":"2024-04-28T12:50:30.869122+0000","last_peered":"2024-04-28T12:50:30.869122+0000","last_clean":"2024-04-28T12:50:30.869122+0000","last_became_active":"2024-04-28T12:48:34.555054+0000","last_became_peered":"2024-04-28T12:48:34.555054+0000","last_unstale":"2024-04-28T12:50:30.869122+0000","last_undegraded":"2024-04-28T12:50:30.869122+0000","last_fullsized":"2024-04-28T12:50:30.869122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:37.493994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858456+0000","last_change":"2024-04-28T12:48:34.568857+0000","last_active":"2024-04-28T12:50:30.858456+0000","last_peered":"2024-04-28T12:50:30.858456+0000","last_clean":"2024-04-28T12:50:30.858456+0000","last_became_active":"2024-04-28T12:48:34.568550+0000","last_became_peered":"2024-04-28T12:48:34.568550+0000","last_unstale":"2024-04-28T12:50:30.858456+0000","last_undegraded":"2024-04-28T12:50:30.858456+0000","last_fullsized":"2024-04-28T12:50:30.858456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:09:40.709446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858104+0000","last_change":"2024-04-28T12:48:39.608021+0000","last_active":"2024-04-28T12:50:30.858104+0000","last_peered":"2024-04-28T12:50:30.858104+0000","last_clean":"2024-04-28T12:50:30.858104+0000","last_became_active":"2024-04-28T12:48:39.607187+0000","last_became_peered":"2024-04-28T12:48:39.607187+0000","last_unstale":"2024-04-28T12:50:30.858104+0000","last_undegraded":"2024-04-28T12:50:30.858104+0000","last_fullsized":"2024-04-28T12:50:30.858104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:31:26.346414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.869351+0000","last_change":"2024-04-28T12:48:33.839179+0000","last_active":"2024-04-28T12:50:30.869351+0000","last_peered":"2024-04-28T12:50:30.869351+0000","last_clean":"2024-04-28T12:50:30.869351+0000","last_became_active":"2024-04-28T12:48:33.838004+0000","last_became_peered":"2024-04-28T12:48:33.838004+0000","last_unstale":"2024-04-28T12:50:30.869351+0000","last_undegraded":"2024-04-28T12:50:30.869351+0000","last_fullsized":"2024-04-28T12:50:30.869351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:32:14.765652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.861376+0000","last_change":"2024-04-28T12:48:34.571840+0000","last_active":"2024-04-28T12:50:30.861376+0000","last_peered":"2024-04-28T12:50:30.861376+0000","last_clean":"2024-04-28T12:50:30.861376+0000","last_became_active":"2024-04-28T12:48:34.568001+0000","last_became_peered":"2024-04-28T12:48:34.568001+0000","last_unstale":"2024-04-28T12:50:30.861376+0000","last_undegraded":"2024-04-28T12:50:30.861376+0000","last_fullsized":"2024-04-28T12:50:30.861376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:15:22.276123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.908708+0000","last_change":"2024-04-28T12:48:39.605055+0000","last_active":"2024-04-28T12:50:30.908708+0000","last_peered":"2024-04-28T12:50:30.908708+0000","last_clean":"2024-04-28T12:50:30.908708+0000","last_became_active":"2024-04-28T12:48:39.604566+0000","last_became_peered":"2024-04-28T12:48:39.604566+0000","last_unstale":"2024-04-28T12:50:30.908708+0000","last_undegraded":"2024-04-28T12:50:30.908708+0000","last_fullsized":"2024-04-28T12:50:30.908708+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:48.945467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.862441+0000","last_change":"2024-04-28T12:48:33.841745+0000","last_active":"2024-04-28T12:50:30.862441+0000","last_peered":"2024-04-28T12:50:30.862441+0000","last_clean":"2024-04-28T12:50:30.862441+0000","last_became_active":"2024-04-28T12:48:33.835003+0000","last_became_peered":"2024-04-28T12:48:33.835003+0000","last_unstale":"2024-04-28T12:50:30.862441+0000","last_undegraded":"2024-04-28T12:50:30.862441+0000","last_fullsized":"2024-04-28T12:50:30.862441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:40:14.253322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.120711+0000","last_change":"2024-04-28T12:48:34.577688+0000","last_active":"2024-04-28T12:50:31.120711+0000","last_peered":"2024-04-28T12:50:31.120711+0000","last_clean":"2024-04-28T12:50:31.120711+0000","last_became_active":"2024-04-28T12:48:34.577269+0000","last_became_peered":"2024-04-28T12:48:34.577269+0000","last_unstale":"2024-04-28T12:50:31.120711+0000","last_undegraded":"2024-04-28T12:50:31.120711+0000","last_fullsized":"2024-04-28T12:50:31.120711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:30:13.744015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.908823+0000","last_change":"2024-04-28T12:48:39.600708+0000","last_active":"2024-04-28T12:50:30.908823+0000","last_peered":"2024-04-28T12:50:30.908823+0000","last_clean":"2024-04-28T12:50:30.908823+0000","last_became_active":"2024-04-28T12:48:39.600257+0000","last_became_peered":"2024-04-28T12:48:39.600257+0000","last_unstale":"2024-04-28T12:50:30.908823+0000","last_undegraded":"2024-04-28T12:50:30.908823+0000","last_fullsized":"2024-04-28T12:50:30.908823+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:08:02.685278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998632+0000","last_change":"2024-04-28T12:48:33.837824+0000","last_active":"2024-04-28T12:50:30.998632+0000","last_peered":"2024-04-28T12:50:30.998632+0000","last_clean":"2024-04-28T12:50:30.998632+0000","last_became_active":"2024-04-28T12:48:33.837505+0000","last_became_peered":"2024-04-28T12:48:33.837505+0000","last_unstale":"2024-04-28T12:50:30.998632+0000","last_undegraded":"2024-04-28T12:50:30.998632+0000","last_fullsized":"2024-04-28T12:50:30.998632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:21:10.677381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.271705+0000","last_change":"2024-04-28T12:48:34.554575+0000","last_active":"2024-04-28T12:50:30.271705+0000","last_peered":"2024-04-28T12:50:30.271705+0000","last_clean":"2024-04-28T12:50:30.271705+0000","last_became_active":"2024-04-28T12:48:34.554214+0000","last_became_peered":"2024-04-28T12:48:34.554214+0000","last_unstale":"2024-04-28T12:50:30.271705+0000","last_undegraded":"2024-04-28T12:50:30.271705+0000","last_fullsized":"2024-04-28T12:50:30.271705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:37:03.369874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.908769+0000","last_change":"2024-04-28T12:48:39.604793+0000","last_active":"2024-04-28T12:50:30.908769+0000","last_peered":"2024-04-28T12:50:30.908769+0000","last_clean":"2024-04-28T12:50:30.908769+0000","last_became_active":"2024-04-28T12:48:39.604338+0000","last_became_peered":"2024-04-28T12:48:39.604338+0000","last_unstale":"2024-04-28T12:50:30.908769+0000","last_undegraded":"2024-04-28T12:50:30.908769+0000","last_fullsized":"2024-04-28T12:50:30.908769+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:14:56.225098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998310+0000","last_change":"2024-04-28T12:48:33.839375+0000","last_active":"2024-04-28T12:50:30.998310+0000","last_peered":"2024-04-28T12:50:30.998310+0000","last_clean":"2024-04-28T12:50:30.998310+0000","last_became_active":"2024-04-28T12:48:33.839062+0000","last_became_peered":"2024-04-28T12:48:33.839062+0000","last_unstale":"2024-04-28T12:50:30.998310+0000","last_undegraded":"2024-04-28T12:50:30.998310+0000","last_fullsized":"2024-04-28T12:50:30.998310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:20:06.692743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.888038+0000","last_change":"2024-04-28T12:48:34.543341+0000","last_active":"2024-04-28T12:50:30.888038+0000","last_peered":"2024-04-28T12:50:30.888038+0000","last_clean":"2024-04-28T12:50:30.888038+0000","last_became_active":"2024-04-28T12:48:34.543033+0000","last_became_peered":"2024-04-28T12:48:34.543033+0000","last_unstale":"2024-04-28T12:50:30.888038+0000","last_undegraded":"2024-04-28T12:50:30.888038+0000","last_fullsized":"2024-04-28T12:50:30.888038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:41:27.654577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858212+0000","last_change":"2024-04-28T12:48:39.589856+0000","last_active":"2024-04-28T12:50:30.858212+0000","last_peered":"2024-04-28T12:50:30.858212+0000","last_clean":"2024-04-28T12:50:30.858212+0000","last_became_active":"2024-04-28T12:48:39.589148+0000","last_became_peered":"2024-04-28T12:48:39.589148+0000","last_unstale":"2024-04-28T12:50:30.858212+0000","last_undegraded":"2024-04-28T12:50:30.858212+0000","last_fullsized":"2024-04-28T12:50:30.858212+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:34:11.897109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":128,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:32.891531+0000","last_change":"2024-04-28T12:48:33.829387+0000","last_active":"2024-04-28T12:50:32.891531+0000","last_peered":"2024-04-28T12:50:32.891531+0000","last_clean":"2024-04-28T12:50:32.891531+0000","last_became_active":"2024-04-28T12:48:33.829011+0000","last_became_peered":"2024-04-28T12:48:33.829011+0000","last_unstale":"2024-04-28T12:50:32.891531+0000","last_undegraded":"2024-04-28T12:50:32.891531+0000","last_fullsized":"2024-04-28T12:50:32.891531+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:54:22.096246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998139+0000","last_change":"2024-04-28T12:48:34.554629+0000","last_active":"2024-04-28T12:50:30.998139+0000","last_peered":"2024-04-28T12:50:30.998139+0000","last_clean":"2024-04-28T12:50:30.998139+0000","last_became_active":"2024-04-28T12:48:34.554231+0000","last_became_peered":"2024-04-28T12:48:34.554231+0000","last_unstale":"2024-04-28T12:50:30.998139+0000","last_undegraded":"2024-04-28T12:50:30.998139+0000","last_fullsized":"2024-04-28T12:50:30.998139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:21:53.706120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858211+0000","last_change":"2024-04-28T12:48:39.606416+0000","last_active":"2024-04-28T12:50:30.858211+0000","last_peered":"2024-04-28T12:50:30.858211+0000","last_clean":"2024-04-28T12:50:30.858211+0000","last_became_active":"2024-04-28T12:48:39.605957+0000","last_became_peered":"2024-04-28T12:48:39.605957+0000","last_unstale":"2024-04-28T12:50:30.858211+0000","last_undegraded":"2024-04-28T12:50:30.858211+0000","last_fullsized":"2024-04-28T12:50:30.858211+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:39:16.315446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.855726+0000","last_change":"2024-04-28T12:48:33.841119+0000","last_active":"2024-04-28T12:50:30.855726+0000","last_peered":"2024-04-28T12:50:30.855726+0000","last_clean":"2024-04-28T12:50:30.855726+0000","last_became_active":"2024-04-28T12:48:33.840833+0000","last_became_peered":"2024-04-28T12:48:33.840833+0000","last_unstale":"2024-04-28T12:50:30.855726+0000","last_undegraded":"2024-04-28T12:50:30.855726+0000","last_fullsized":"2024-04-28T12:50:30.855726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:49:31.145968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887846+0000","last_change":"2024-04-28T12:48:34.565140+0000","last_active":"2024-04-28T12:50:30.887846+0000","last_peered":"2024-04-28T12:50:30.887846+0000","last_clean":"2024-04-28T12:50:30.887846+0000","last_became_active":"2024-04-28T12:48:34.564692+0000","last_became_peered":"2024-04-28T12:48:34.564692+0000","last_unstale":"2024-04-28T12:50:30.887846+0000","last_undegraded":"2024-04-28T12:50:30.887846+0000","last_fullsized":"2024-04-28T12:50:30.887846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:55:27.440482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117850+0000","last_change":"2024-04-28T12:48:39.611109+0000","last_active":"2024-04-28T12:50:31.117850+0000","last_peered":"2024-04-28T12:50:31.117850+0000","last_clean":"2024-04-28T12:50:31.117850+0000","last_became_active":"2024-04-28T12:48:39.610754+0000","last_became_peered":"2024-04-28T12:48:39.610754+0000","last_unstale":"2024-04-28T12:50:31.117850+0000","last_undegraded":"2024-04-28T12:50:31.117850+0000","last_fullsized":"2024-04-28T12:50:31.117850+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:55:14.610718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.906082+0000","last_change":"2024-04-28T12:48:33.838037+0000","last_active":"2024-04-28T12:50:30.906082+0000","last_peered":"2024-04-28T12:50:30.906082+0000","last_clean":"2024-04-28T12:50:30.906082+0000","last_became_active":"2024-04-28T12:48:33.837704+0000","last_became_peered":"2024-04-28T12:48:33.837704+0000","last_unstale":"2024-04-28T12:50:30.906082+0000","last_undegraded":"2024-04-28T12:50:30.906082+0000","last_fullsized":"2024-04-28T12:50:30.906082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:46:24.595855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.123235+0000","last_change":"2024-04-28T12:50:31.120476+0000","last_active":"2024-04-28T12:50:31.123235+0000","last_peered":"2024-04-28T12:50:31.123235+0000","last_clean":"2024-04-28T12:50:31.123235+0000","last_became_active":"2024-04-28T12:50:31.119338+0000","last_became_peered":"2024-04-28T12:50:31.119338+0000","last_unstale":"2024-04-28T12:50:31.123235+0000","last_undegraded":"2024-04-28T12:50:31.123235+0000","last_fullsized":"2024-04-28T12:50:31.123235+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:35:07.965270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.120110+0000","last_change":"2024-04-28T12:48:39.615515+0000","last_active":"2024-04-28T12:50:31.120110+0000","last_peered":"2024-04-28T12:50:31.120110+0000","last_clean":"2024-04-28T12:50:31.120110+0000","last_became_active":"2024-04-28T12:48:39.614717+0000","last_became_peered":"2024-04-28T12:48:39.614717+0000","last_unstale":"2024-04-28T12:50:31.120110+0000","last_undegraded":"2024-04-28T12:50:31.120110+0000","last_fullsized":"2024-04-28T12:50:31.120110+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:11:06.633399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887427+0000","last_change":"2024-04-28T12:48:33.832694+0000","last_active":"2024-04-28T12:50:30.887427+0000","last_peered":"2024-04-28T12:50:30.887427+0000","last_clean":"2024-04-28T12:50:30.887427+0000","last_became_active":"2024-04-28T12:48:33.832381+0000","last_became_peered":"2024-04-28T12:48:33.832381+0000","last_unstale":"2024-04-28T12:50:30.887427+0000","last_undegraded":"2024-04-28T12:50:30.887427+0000","last_fullsized":"2024-04-28T12:50:30.887427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:56:57.511053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.865713+0000","last_change":"2024-04-28T12:48:34.560229+0000","last_active":"2024-04-28T12:50:30.865713+0000","last_peered":"2024-04-28T12:50:30.865713+0000","last_clean":"2024-04-28T12:50:30.865713+0000","last_became_active":"2024-04-28T12:48:34.559926+0000","last_became_peered":"2024-04-28T12:48:34.559926+0000","last_unstale":"2024-04-28T12:50:30.865713+0000","last_undegraded":"2024-04-28T12:50:30.865713+0000","last_fullsized":"2024-04-28T12:50:30.865713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:38:22.476449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.860154+0000","last_change":"2024-04-28T12:48:39.604042+0000","last_active":"2024-04-28T12:50:30.860154+0000","last_peered":"2024-04-28T12:50:30.860154+0000","last_clean":"2024-04-28T12:50:30.860154+0000","last_became_active":"2024-04-28T12:48:39.603631+0000","last_became_peered":"2024-04-28T12:48:39.603631+0000","last_unstale":"2024-04-28T12:50:30.860154+0000","last_undegraded":"2024-04-28T12:50:30.860154+0000","last_fullsized":"2024-04-28T12:50:30.860154+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:53:20.073434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":121,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:32.890850+0000","last_change":"2024-04-28T12:48:33.846372+0000","last_active":"2024-04-28T12:50:32.890850+0000","last_peered":"2024-04-28T12:50:32.890850+0000","last_clean":"2024-04-28T12:50:32.890850+0000","last_became_active":"2024-04-28T12:48:33.846138+0000","last_became_peered":"2024-04-28T12:48:33.846138+0000","last_unstale":"2024-04-28T12:50:32.890850+0000","last_undegraded":"2024-04-28T12:50:32.890850+0000","last_fullsized":"2024-04-28T12:50:32.890850+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:46:45.889188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.272120+0000","last_change":"2024-04-28T12:48:34.554872+0000","last_active":"2024-04-28T12:50:30.272120+0000","last_peered":"2024-04-28T12:50:30.272120+0000","last_clean":"2024-04-28T12:50:30.272120+0000","last_became_active":"2024-04-28T12:48:34.554555+0000","last_became_peered":"2024-04-28T12:48:34.554555+0000","last_unstale":"2024-04-28T12:50:30.272120+0000","last_undegraded":"2024-04-28T12:50:30.272120+0000","last_fullsized":"2024-04-28T12:50:30.272120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:17:51.280686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.866600+0000","last_change":"2024-04-28T12:48:39.603114+0000","last_active":"2024-04-28T12:50:30.866600+0000","last_peered":"2024-04-28T12:50:30.866600+0000","last_clean":"2024-04-28T12:50:30.866600+0000","last_became_active":"2024-04-28T12:48:39.598404+0000","last_became_peered":"2024-04-28T12:48:39.598404+0000","last_unstale":"2024-04-28T12:50:30.866600+0000","last_undegraded":"2024-04-28T12:50:30.866600+0000","last_fullsized":"2024-04-28T12:50:30.866600+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:23.824317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118002+0000","last_change":"2024-04-28T12:48:33.831626+0000","last_active":"2024-04-28T12:50:31.118002+0000","last_peered":"2024-04-28T12:50:31.118002+0000","last_clean":"2024-04-28T12:50:31.118002+0000","last_became_active":"2024-04-28T12:48:33.831294+0000","last_became_peered":"2024-04-28T12:48:33.831294+0000","last_unstale":"2024-04-28T12:50:31.118002+0000","last_undegraded":"2024-04-28T12:50:31.118002+0000","last_fullsized":"2024-04-28T12:50:31.118002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:58:43.775695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.120242+0000","last_change":"2024-04-28T12:48:34.574987+0000","last_active":"2024-04-28T12:50:31.120242+0000","last_peered":"2024-04-28T12:50:31.120242+0000","last_clean":"2024-04-28T12:50:31.120242+0000","last_became_active":"2024-04-28T12:48:34.574668+0000","last_became_peered":"2024-04-28T12:48:34.574668+0000","last_unstale":"2024-04-28T12:50:31.120242+0000","last_undegraded":"2024-04-28T12:50:31.120242+0000","last_fullsized":"2024-04-28T12:50:31.120242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:24:06.195671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.908966+0000","last_change":"2024-04-28T12:49:30.994058+0000","last_active":"2024-04-28T12:50:30.908966+0000","last_peered":"2024-04-28T12:50:30.908966+0000","last_clean":"2024-04-28T12:50:30.908966+0000","last_became_active":"2024-04-28T12:49:30.993311+0000","last_became_peered":"2024-04-28T12:49:30.993311+0000","last_unstale":"2024-04-28T12:50:30.908966+0000","last_undegraded":"2024-04-28T12:50:30.908966+0000","last_fullsized":"2024-04-28T12:50:30.908966+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:37:08.699751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.888036+0000","last_change":"2024-04-28T12:48:33.840884+0000","last_active":"2024-04-28T12:50:30.888036+0000","last_peered":"2024-04-28T12:50:30.888036+0000","last_clean":"2024-04-28T12:50:30.888036+0000","last_became_active":"2024-04-28T12:48:33.840467+0000","last_became_peered":"2024-04-28T12:48:33.840467+0000","last_unstale":"2024-04-28T12:50:30.888036+0000","last_undegraded":"2024-04-28T12:50:30.888036+0000","last_fullsized":"2024-04-28T12:50:30.888036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:28:16.480520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.271817+0000","last_change":"2024-04-28T12:48:34.545000+0000","last_active":"2024-04-28T12:50:30.271817+0000","last_peered":"2024-04-28T12:50:30.271817+0000","last_clean":"2024-04-28T12:50:30.271817+0000","last_became_active":"2024-04-28T12:48:34.544662+0000","last_became_peered":"2024-04-28T12:48:34.544662+0000","last_unstale":"2024-04-28T12:50:30.271817+0000","last_undegraded":"2024-04-28T12:50:30.271817+0000","last_fullsized":"2024-04-28T12:50:30.271817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:19:21.640362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.866287+0000","last_change":"2024-04-28T12:48:39.596755+0000","last_active":"2024-04-28T12:50:30.866287+0000","last_peered":"2024-04-28T12:50:30.866287+0000","last_clean":"2024-04-28T12:50:30.866287+0000","last_became_active":"2024-04-28T12:48:39.596292+0000","last_became_peered":"2024-04-28T12:48:39.596292+0000","last_unstale":"2024-04-28T12:50:30.866287+0000","last_undegraded":"2024-04-28T12:50:30.866287+0000","last_fullsized":"2024-04-28T12:50:30.866287+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:21:27.026940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'32","reported_seq":244,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:33.892088+0000","last_change":"2024-04-28T12:48:33.833840+0000","last_active":"2024-04-28T12:50:33.892088+0000","last_peered":"2024-04-28T12:50:33.892088+0000","last_clean":"2024-04-28T12:50:33.892088+0000","last_became_active":"2024-04-28T12:48:33.833536+0000","last_became_peered":"2024-04-28T12:48:33.833536+0000","last_unstale":"2024-04-28T12:50:33.892088+0000","last_undegraded":"2024-04-28T12:50:33.892088+0000","last_fullsized":"2024-04-28T12:50:33.892088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:37:01.455373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54826,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":180,"num_read_kb":143,"num_write":33,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998494+0000","last_change":"2024-04-28T12:48:34.548046+0000","last_active":"2024-04-28T12:50:30.998494+0000","last_peered":"2024-04-28T12:50:30.998494+0000","last_clean":"2024-04-28T12:50:30.998494+0000","last_became_active":"2024-04-28T12:48:34.547724+0000","last_became_peered":"2024-04-28T12:48:34.547724+0000","last_unstale":"2024-04-28T12:50:30.998494+0000","last_undegraded":"2024-04-28T12:50:30.998494+0000","last_fullsized":"2024-04-28T12:50:30.998494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:07.287597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.888331+0000","last_change":"2024-04-28T12:48:39.602976+0000","last_active":"2024-04-28T12:50:30.888331+0000","last_peered":"2024-04-28T12:50:30.888331+0000","last_clean":"2024-04-28T12:50:30.888331+0000","last_became_active":"2024-04-28T12:48:39.602588+0000","last_became_peered":"2024-04-28T12:48:39.602588+0000","last_unstale":"2024-04-28T12:50:30.888331+0000","last_undegraded":"2024-04-28T12:50:30.888331+0000","last_fullsized":"2024-04-28T12:50:30.888331+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:18:19.549434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858203+0000","last_change":"2024-04-28T12:48:33.831276+0000","last_active":"2024-04-28T12:50:30.858203+0000","last_peered":"2024-04-28T12:50:30.858203+0000","last_clean":"2024-04-28T12:50:30.858203+0000","last_became_active":"2024-04-28T12:48:33.830845+0000","last_became_peered":"2024-04-28T12:48:33.830845+0000","last_unstale":"2024-04-28T12:50:30.858203+0000","last_undegraded":"2024-04-28T12:50:30.858203+0000","last_fullsized":"2024-04-28T12:50:30.858203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:21:02.619309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.905332+0000","last_change":"2024-04-28T12:48:34.566767+0000","last_active":"2024-04-28T12:50:30.905332+0000","last_peered":"2024-04-28T12:50:30.905332+0000","last_clean":"2024-04-28T12:50:30.905332+0000","last_became_active":"2024-04-28T12:48:34.566404+0000","last_became_peered":"2024-04-28T12:48:34.566404+0000","last_unstale":"2024-04-28T12:50:30.905332+0000","last_undegraded":"2024-04-28T12:50:30.905332+0000","last_fullsized":"2024-04-28T12:50:30.905332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:49:32.684046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117616+0000","last_change":"2024-04-28T12:48:39.607471+0000","last_active":"2024-04-28T12:50:31.117616+0000","last_peered":"2024-04-28T12:50:31.117616+0000","last_clean":"2024-04-28T12:50:31.117616+0000","last_became_active":"2024-04-28T12:48:39.607040+0000","last_became_peered":"2024-04-28T12:48:39.607040+0000","last_unstale":"2024-04-28T12:50:31.117616+0000","last_undegraded":"2024-04-28T12:50:31.117616+0000","last_fullsized":"2024-04-28T12:50:31.117616+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:06:32.613106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887844+0000","last_change":"2024-04-28T12:48:33.832519+0000","last_active":"2024-04-28T12:50:30.887844+0000","last_peered":"2024-04-28T12:50:30.887844+0000","last_clean":"2024-04-28T12:50:30.887844+0000","last_became_active":"2024-04-28T12:48:33.832146+0000","last_became_peered":"2024-04-28T12:48:33.832146+0000","last_unstale":"2024-04-28T12:50:30.887844+0000","last_undegraded":"2024-04-28T12:50:30.887844+0000","last_fullsized":"2024-04-28T12:50:30.887844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:02:48.755990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118096+0000","last_change":"2024-04-28T12:48:34.557093+0000","last_active":"2024-04-28T12:50:31.118096+0000","last_peered":"2024-04-28T12:50:31.118096+0000","last_clean":"2024-04-28T12:50:31.118096+0000","last_became_active":"2024-04-28T12:48:34.556767+0000","last_became_peered":"2024-04-28T12:48:34.556767+0000","last_unstale":"2024-04-28T12:50:31.118096+0000","last_undegraded":"2024-04-28T12:50:31.118096+0000","last_fullsized":"2024-04-28T12:50:31.118096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:45:48.634998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.868913+0000","last_change":"2024-04-28T12:48:39.612227+0000","last_active":"2024-04-28T12:50:30.868913+0000","last_peered":"2024-04-28T12:50:30.868913+0000","last_clean":"2024-04-28T12:50:30.868913+0000","last_became_active":"2024-04-28T12:48:39.611065+0000","last_became_peered":"2024-04-28T12:48:39.611065+0000","last_unstale":"2024-04-28T12:50:30.868913+0000","last_undegraded":"2024-04-28T12:50:30.868913+0000","last_fullsized":"2024-04-28T12:50:30.868913+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:15:55.833369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":127,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:33.915057+0000","last_change":"2024-04-28T12:48:33.835102+0000","last_active":"2024-04-28T12:50:33.915057+0000","last_peered":"2024-04-28T12:50:33.915057+0000","last_clean":"2024-04-28T12:50:33.915057+0000","last_became_active":"2024-04-28T12:48:33.834672+0000","last_became_peered":"2024-04-28T12:48:33.834672+0000","last_unstale":"2024-04-28T12:50:33.915057+0000","last_undegraded":"2024-04-28T12:50:33.915057+0000","last_fullsized":"2024-04-28T12:50:33.915057+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:41:39.388880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117740+0000","last_change":"2024-04-28T12:49:30.707963+0000","last_active":"2024-04-28T12:50:31.117740+0000","last_peered":"2024-04-28T12:50:31.117740+0000","last_clean":"2024-04-28T12:50:31.117740+0000","last_became_active":"2024-04-28T12:49:30.707565+0000","last_became_peered":"2024-04-28T12:49:30.707565+0000","last_unstale":"2024-04-28T12:50:31.117740+0000","last_undegraded":"2024-04-28T12:50:31.117740+0000","last_fullsized":"2024-04-28T12:50:31.117740+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:44:34.238749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.906986+0000","last_change":"2024-04-28T12:48:39.613039+0000","last_active":"2024-04-28T12:50:30.906986+0000","last_peered":"2024-04-28T12:50:30.906986+0000","last_clean":"2024-04-28T12:50:30.906986+0000","last_became_active":"2024-04-28T12:48:39.609383+0000","last_became_peered":"2024-04-28T12:48:39.609383+0000","last_unstale":"2024-04-28T12:50:30.906986+0000","last_undegraded":"2024-04-28T12:50:30.906986+0000","last_fullsized":"2024-04-28T12:50:30.906986+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:55:47.031973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998022+0000","last_change":"2024-04-28T12:48:33.843087+0000","last_active":"2024-04-28T12:50:30.998022+0000","last_peered":"2024-04-28T12:50:30.998022+0000","last_clean":"2024-04-28T12:50:30.998022+0000","last_became_active":"2024-04-28T12:48:33.842775+0000","last_became_peered":"2024-04-28T12:48:33.842775+0000","last_unstale":"2024-04-28T12:50:30.998022+0000","last_undegraded":"2024-04-28T12:50:30.998022+0000","last_fullsized":"2024-04-28T12:50:30.998022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:39:00.797781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887689+0000","last_change":"2024-04-28T12:49:30.993703+0000","last_active":"2024-04-28T12:50:30.887689+0000","last_peered":"2024-04-28T12:50:30.887689+0000","last_clean":"2024-04-28T12:50:30.887689+0000","last_became_active":"2024-04-28T12:49:30.992940+0000","last_became_peered":"2024-04-28T12:49:30.992940+0000","last_unstale":"2024-04-28T12:50:30.887689+0000","last_undegraded":"2024-04-28T12:50:30.887689+0000","last_fullsized":"2024-04-28T12:50:30.887689+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:42:01.384945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887656+0000","last_change":"2024-04-28T12:48:39.608191+0000","last_active":"2024-04-28T12:50:30.887656+0000","last_peered":"2024-04-28T12:50:30.887656+0000","last_clean":"2024-04-28T12:50:30.887656+0000","last_became_active":"2024-04-28T12:48:39.607724+0000","last_became_peered":"2024-04-28T12:48:39.607724+0000","last_unstale":"2024-04-28T12:50:30.887656+0000","last_undegraded":"2024-04-28T12:50:30.887656+0000","last_fullsized":"2024-04-28T12:50:30.887656+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:19:58.040666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998502+0000","last_change":"2024-04-28T12:48:33.839588+0000","last_active":"2024-04-28T12:50:30.998502+0000","last_peered":"2024-04-28T12:50:30.998502+0000","last_clean":"2024-04-28T12:50:30.998502+0000","last_became_active":"2024-04-28T12:48:33.839094+0000","last_became_peered":"2024-04-28T12:48:33.839094+0000","last_unstale":"2024-04-28T12:50:30.998502+0000","last_undegraded":"2024-04-28T12:50:30.998502+0000","last_fullsized":"2024-04-28T12:50:30.998502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:18:35.824206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118047+0000","last_change":"2024-04-28T12:49:30.791511+0000","last_active":"2024-04-28T12:50:31.118047+0000","last_peered":"2024-04-28T12:50:31.118047+0000","last_clean":"2024-04-28T12:50:31.118047+0000","last_became_active":"2024-04-28T12:49:30.791098+0000","last_became_peered":"2024-04-28T12:49:30.791098+0000","last_unstale":"2024-04-28T12:50:31.118047+0000","last_undegraded":"2024-04-28T12:50:31.118047+0000","last_fullsized":"2024-04-28T12:50:31.118047+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:53:02.628572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998138+0000","last_change":"2024-04-28T12:48:39.912611+0000","last_active":"2024-04-28T12:50:30.998138+0000","last_peered":"2024-04-28T12:50:30.998138+0000","last_clean":"2024-04-28T12:50:30.998138+0000","last_became_active":"2024-04-28T12:48:39.912097+0000","last_became_peered":"2024-04-28T12:48:39.912097+0000","last_unstale":"2024-04-28T12:50:30.998138+0000","last_undegraded":"2024-04-28T12:50:30.998138+0000","last_fullsized":"2024-04-28T12:50:30.998138+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:31:38.597920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.867921+0000","last_change":"2024-04-28T12:49:30.708298+0000","last_active":"2024-04-28T12:50:30.867921+0000","last_peered":"2024-04-28T12:50:30.867921+0000","last_clean":"2024-04-28T12:50:30.867921+0000","last_became_active":"2024-04-28T12:49:30.707831+0000","last_became_peered":"2024-04-28T12:49:30.707831+0000","last_unstale":"2024-04-28T12:50:30.867921+0000","last_undegraded":"2024-04-28T12:50:30.867921+0000","last_fullsized":"2024-04-28T12:50:30.867921+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:09:21.049858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.856183+0000","last_change":"2024-04-28T12:48:34.568892+0000","last_active":"2024-04-28T12:50:30.856183+0000","last_peered":"2024-04-28T12:50:30.856183+0000","last_clean":"2024-04-28T12:50:30.856183+0000","last_became_active":"2024-04-28T12:48:34.568581+0000","last_became_peered":"2024-04-28T12:48:34.568581+0000","last_unstale":"2024-04-28T12:50:30.856183+0000","last_undegraded":"2024-04-28T12:50:30.856183+0000","last_fullsized":"2024-04-28T12:50:30.856183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:56:51.661049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.906256+0000","last_change":"2024-04-28T12:48:39.616095+0000","last_active":"2024-04-28T12:50:30.906256+0000","last_peered":"2024-04-28T12:50:30.906256+0000","last_clean":"2024-04-28T12:50:30.906256+0000","last_became_active":"2024-04-28T12:48:39.614891+0000","last_became_peered":"2024-04-28T12:48:39.614891+0000","last_unstale":"2024-04-28T12:50:30.906256+0000","last_undegraded":"2024-04-28T12:50:30.906256+0000","last_fullsized":"2024-04-28T12:50:30.906256+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:01:05.645329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117867+0000","last_change":"2024-04-28T12:48:33.833599+0000","last_active":"2024-04-28T12:50:31.117867+0000","last_peered":"2024-04-28T12:50:31.117867+0000","last_clean":"2024-04-28T12:50:31.117867+0000","last_became_active":"2024-04-28T12:48:33.833062+0000","last_became_peered":"2024-04-28T12:48:33.833062+0000","last_unstale":"2024-04-28T12:50:31.117867+0000","last_undegraded":"2024-04-28T12:50:31.117867+0000","last_fullsized":"2024-04-28T12:50:31.117867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:00:13.094007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.855953+0000","last_change":"2024-04-28T12:48:34.558759+0000","last_active":"2024-04-28T12:50:30.855953+0000","last_peered":"2024-04-28T12:50:30.855953+0000","last_clean":"2024-04-28T12:50:30.855953+0000","last_became_active":"2024-04-28T12:48:34.558413+0000","last_became_peered":"2024-04-28T12:48:34.558413+0000","last_unstale":"2024-04-28T12:50:30.855953+0000","last_undegraded":"2024-04-28T12:50:30.855953+0000","last_fullsized":"2024-04-28T12:50:30.855953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:07:47.106653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.907209+0000","last_change":"2024-04-28T12:48:39.914492+0000","last_active":"2024-04-28T12:50:30.907209+0000","last_peered":"2024-04-28T12:50:30.907209+0000","last_clean":"2024-04-28T12:50:30.907209+0000","last_became_active":"2024-04-28T12:48:39.913113+0000","last_became_peered":"2024-04-28T12:48:39.913113+0000","last_unstale":"2024-04-28T12:50:30.907209+0000","last_undegraded":"2024-04-28T12:50:30.907209+0000","last_fullsized":"2024-04-28T12:50:30.907209+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:49:21.221058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.888040+0000","last_change":"2024-04-28T12:48:34.544121+0000","last_active":"2024-04-28T12:50:30.888040+0000","last_peered":"2024-04-28T12:50:30.888040+0000","last_clean":"2024-04-28T12:50:30.888040+0000","last_became_active":"2024-04-28T12:48:34.543816+0000","last_became_peered":"2024-04-28T12:48:34.543816+0000","last_unstale":"2024-04-28T12:50:30.888040+0000","last_undegraded":"2024-04-28T12:50:30.888040+0000","last_fullsized":"2024-04-28T12:50:30.888040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:24.237518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":123,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:33.914392+0000","last_change":"2024-04-28T12:48:33.845996+0000","last_active":"2024-04-28T12:50:33.914392+0000","last_peered":"2024-04-28T12:50:33.914392+0000","last_clean":"2024-04-28T12:50:33.914392+0000","last_became_active":"2024-04-28T12:48:33.845618+0000","last_became_peered":"2024-04-28T12:48:33.845618+0000","last_unstale":"2024-04-28T12:50:33.914392+0000","last_undegraded":"2024-04-28T12:50:33.914392+0000","last_fullsized":"2024-04-28T12:50:33.914392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:53:12.312461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117959+0000","last_change":"2024-04-28T12:48:39.589281+0000","last_active":"2024-04-28T12:50:31.117959+0000","last_peered":"2024-04-28T12:50:31.117959+0000","last_clean":"2024-04-28T12:50:31.117959+0000","last_became_active":"2024-04-28T12:48:39.588964+0000","last_became_peered":"2024-04-28T12:48:39.588964+0000","last_unstale":"2024-04-28T12:50:31.117959+0000","last_undegraded":"2024-04-28T12:50:31.117959+0000","last_fullsized":"2024-04-28T12:50:31.117959+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:32:12.039897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.860817+0000","last_change":"2024-04-28T12:48:34.567062+0000","last_active":"2024-04-28T12:50:30.860817+0000","last_peered":"2024-04-28T12:50:30.860817+0000","last_clean":"2024-04-28T12:50:30.860817+0000","last_became_active":"2024-04-28T12:48:34.565977+0000","last_became_peered":"2024-04-28T12:48:34.565977+0000","last_unstale":"2024-04-28T12:50:30.860817+0000","last_undegraded":"2024-04-28T12:50:30.860817+0000","last_fullsized":"2024-04-28T12:50:30.860817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:46:03.775693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.908912+0000","last_change":"2024-04-28T12:48:33.834069+0000","last_active":"2024-04-28T12:50:30.908912+0000","last_peered":"2024-04-28T12:50:30.908912+0000","last_clean":"2024-04-28T12:50:30.908912+0000","last_became_active":"2024-04-28T12:48:33.833777+0000","last_became_peered":"2024-04-28T12:48:33.833777+0000","last_unstale":"2024-04-28T12:50:30.908912+0000","last_undegraded":"2024-04-28T12:50:30.908912+0000","last_fullsized":"2024-04-28T12:50:30.908912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:32:52.737718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":356,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117523+0000","last_change":"2024-04-28T12:47:13.177589+0000","last_active":"2024-04-28T12:50:31.117523+0000","last_peered":"2024-04-28T12:50:31.117523+0000","last_clean":"2024-04-28T12:50:31.117523+0000","last_became_active":"2024-04-28T12:47:13.177133+0000","last_became_peered":"2024-04-28T12:47:13.177133+0000","last_unstale":"2024-04-28T12:50:31.117523+0000","last_undegraded":"2024-04-28T12:50:31.117523+0000","last_fullsized":"2024-04-28T12:50:31.117523+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:43:15.851674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:43:15.851674+0000","last_clean_scrub_stamp":"2024-04-28T12:43:15.851674+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:23:21.868028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887921+0000","last_change":"2024-04-28T12:49:30.791968+0000","last_active":"2024-04-28T12:50:30.887921+0000","last_peered":"2024-04-28T12:50:30.887921+0000","last_clean":"2024-04-28T12:50:30.887921+0000","last_became_active":"2024-04-28T12:49:30.791380+0000","last_became_peered":"2024-04-28T12:49:30.791380+0000","last_unstale":"2024-04-28T12:50:30.887921+0000","last_undegraded":"2024-04-28T12:50:30.887921+0000","last_fullsized":"2024-04-28T12:50:30.887921+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:56:31.784446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.905166+0000","last_change":"2024-04-28T12:48:34.550976+0000","last_active":"2024-04-28T12:50:30.905166+0000","last_peered":"2024-04-28T12:50:30.905166+0000","last_clean":"2024-04-28T12:50:30.905166+0000","last_became_active":"2024-04-28T12:48:34.549460+0000","last_became_peered":"2024-04-28T12:48:34.549460+0000","last_unstale":"2024-04-28T12:50:30.905166+0000","last_undegraded":"2024-04-28T12:50:30.905166+0000","last_fullsized":"2024-04-28T12:50:30.905166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:55:16.272917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117649+0000","last_change":"2024-04-28T12:48:33.837248+0000","last_active":"2024-04-28T12:50:31.117649+0000","last_peered":"2024-04-28T12:50:31.117649+0000","last_clean":"2024-04-28T12:50:31.117649+0000","last_became_active":"2024-04-28T12:48:33.836910+0000","last_became_peered":"2024-04-28T12:48:33.836910+0000","last_unstale":"2024-04-28T12:50:31.117649+0000","last_undegraded":"2024-04-28T12:50:31.117649+0000","last_fullsized":"2024-04-28T12:50:31.117649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:06:19.572724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117568+0000","last_change":"2024-04-28T12:48:39.605436+0000","last_active":"2024-04-28T12:50:31.117568+0000","last_peered":"2024-04-28T12:50:31.117568+0000","last_clean":"2024-04-28T12:50:31.117568+0000","last_became_active":"2024-04-28T12:48:39.604997+0000","last_became_peered":"2024-04-28T12:48:39.604997+0000","last_unstale":"2024-04-28T12:50:31.117568+0000","last_undegraded":"2024-04-28T12:50:31.117568+0000","last_fullsized":"2024-04-28T12:50:31.117568+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:05:11.571064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998502+0000","last_change":"2024-04-28T12:48:33.848257+0000","last_active":"2024-04-28T12:50:30.998502+0000","last_peered":"2024-04-28T12:50:30.998502+0000","last_clean":"2024-04-28T12:50:30.998502+0000","last_became_active":"2024-04-28T12:48:33.848066+0000","last_became_peered":"2024-04-28T12:48:33.848066+0000","last_unstale":"2024-04-28T12:50:30.998502+0000","last_undegraded":"2024-04-28T12:50:30.998502+0000","last_fullsized":"2024-04-28T12:50:30.998502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:58:34.174153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.907075+0000","last_change":"2024-04-28T12:48:34.559791+0000","last_active":"2024-04-28T12:50:30.907075+0000","last_peered":"2024-04-28T12:50:30.907075+0000","last_clean":"2024-04-28T12:50:30.907075+0000","last_became_active":"2024-04-28T12:48:34.556980+0000","last_became_peered":"2024-04-28T12:48:34.556980+0000","last_unstale":"2024-04-28T12:50:30.907075+0000","last_undegraded":"2024-04-28T12:50:30.907075+0000","last_fullsized":"2024-04-28T12:50:30.907075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:38:56.998159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.860153+0000","last_change":"2024-04-28T12:48:39.917423+0000","last_active":"2024-04-28T12:50:30.860153+0000","last_peered":"2024-04-28T12:50:30.860153+0000","last_clean":"2024-04-28T12:50:30.860153+0000","last_became_active":"2024-04-28T12:48:39.914893+0000","last_became_peered":"2024-04-28T12:48:39.914893+0000","last_unstale":"2024-04-28T12:50:30.860153+0000","last_undegraded":"2024-04-28T12:50:30.860153+0000","last_fullsized":"2024-04-28T12:50:30.860153+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:08:05.499808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'9","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.869240+0000","last_change":"2024-04-28T12:49:30.715567+0000","last_active":"2024-04-28T12:50:30.869240+0000","last_peered":"2024-04-28T12:50:30.869240+0000","last_clean":"2024-04-28T12:50:30.869240+0000","last_became_active":"2024-04-28T12:49:30.711050+0000","last_became_peered":"2024-04-28T12:49:30.711050+0000","last_unstale":"2024-04-28T12:50:30.869240+0000","last_undegraded":"2024-04-28T12:50:30.869240+0000","last_fullsized":"2024-04-28T12:50:30.869240+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:01:32.936140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.863738+0000","last_change":"2024-04-28T12:48:34.566120+0000","last_active":"2024-04-28T12:50:30.863738+0000","last_peered":"2024-04-28T12:50:30.863738+0000","last_clean":"2024-04-28T12:50:30.863738+0000","last_became_active":"2024-04-28T12:48:34.564357+0000","last_became_peered":"2024-04-28T12:48:34.564357+0000","last_unstale":"2024-04-28T12:50:30.863738+0000","last_undegraded":"2024-04-28T12:50:30.863738+0000","last_fullsized":"2024-04-28T12:50:30.863738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:28:43.463008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.857836+0000","last_change":"2024-04-28T12:48:39.586362+0000","last_active":"2024-04-28T12:50:30.857836+0000","last_peered":"2024-04-28T12:50:30.857836+0000","last_clean":"2024-04-28T12:50:30.857836+0000","last_became_active":"2024-04-28T12:48:39.585898+0000","last_became_peered":"2024-04-28T12:48:39.585898+0000","last_unstale":"2024-04-28T12:50:30.857836+0000","last_undegraded":"2024-04-28T12:50:30.857836+0000","last_fullsized":"2024-04-28T12:50:30.857836+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:51:19.098797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.909323+0000","last_change":"2024-04-28T12:48:33.833163+0000","last_active":"2024-04-28T12:50:30.909323+0000","last_peered":"2024-04-28T12:50:30.909323+0000","last_clean":"2024-04-28T12:50:30.909323+0000","last_became_active":"2024-04-28T12:48:33.832798+0000","last_became_peered":"2024-04-28T12:48:33.832798+0000","last_unstale":"2024-04-28T12:50:30.909323+0000","last_undegraded":"2024-04-28T12:50:30.909323+0000","last_fullsized":"2024-04-28T12:50:30.909323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:42:07.511346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.120766+0000","last_change":"2024-04-28T12:48:34.576917+0000","last_active":"2024-04-28T12:50:31.120766+0000","last_peered":"2024-04-28T12:50:31.120766+0000","last_clean":"2024-04-28T12:50:31.120766+0000","last_became_active":"2024-04-28T12:48:34.575602+0000","last_became_peered":"2024-04-28T12:48:34.575602+0000","last_unstale":"2024-04-28T12:50:31.120766+0000","last_undegraded":"2024-04-28T12:50:31.120766+0000","last_fullsized":"2024-04-28T12:50:31.120766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:47:21.769208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.909137+0000","last_change":"2024-04-28T12:48:39.913164+0000","last_active":"2024-04-28T12:50:30.909137+0000","last_peered":"2024-04-28T12:50:30.909137+0000","last_clean":"2024-04-28T12:50:30.909137+0000","last_became_active":"2024-04-28T12:48:39.912716+0000","last_became_peered":"2024-04-28T12:48:39.912716+0000","last_unstale":"2024-04-28T12:50:30.909137+0000","last_undegraded":"2024-04-28T12:50:30.909137+0000","last_fullsized":"2024-04-28T12:50:30.909137+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:26:34.544275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.271370+0000","last_change":"2024-04-28T12:48:33.831286+0000","last_active":"2024-04-28T12:50:30.271370+0000","last_peered":"2024-04-28T12:50:30.271370+0000","last_clean":"2024-04-28T12:50:30.271370+0000","last_became_active":"2024-04-28T12:48:33.830846+0000","last_became_peered":"2024-04-28T12:48:33.830846+0000","last_unstale":"2024-04-28T12:50:30.271370+0000","last_undegraded":"2024-04-28T12:50:30.271370+0000","last_fullsized":"2024-04-28T12:50:30.271370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:32:47.985987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998044+0000","last_change":"2024-04-28T12:48:34.543780+0000","last_active":"2024-04-28T12:50:30.998044+0000","last_peered":"2024-04-28T12:50:30.998044+0000","last_clean":"2024-04-28T12:50:30.998044+0000","last_became_active":"2024-04-28T12:48:34.543441+0000","last_became_peered":"2024-04-28T12:48:34.543441+0000","last_unstale":"2024-04-28T12:50:30.998044+0000","last_undegraded":"2024-04-28T12:50:30.998044+0000","last_fullsized":"2024-04-28T12:50:30.998044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:05:43.334523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.869327+0000","last_change":"2024-04-28T12:48:39.607360+0000","last_active":"2024-04-28T12:50:30.869327+0000","last_peered":"2024-04-28T12:50:30.869327+0000","last_clean":"2024-04-28T12:50:30.869327+0000","last_became_active":"2024-04-28T12:48:39.606944+0000","last_became_peered":"2024-04-28T12:48:39.606944+0000","last_unstale":"2024-04-28T12:50:30.869327+0000","last_undegraded":"2024-04-28T12:50:30.869327+0000","last_fullsized":"2024-04-28T12:50:30.869327+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:59:15.020193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.860612+0000","last_change":"2024-04-28T12:48:33.835694+0000","last_active":"2024-04-28T12:50:30.860612+0000","last_peered":"2024-04-28T12:50:30.860612+0000","last_clean":"2024-04-28T12:50:30.860612+0000","last_became_active":"2024-04-28T12:48:33.835210+0000","last_became_peered":"2024-04-28T12:48:33.835210+0000","last_unstale":"2024-04-28T12:50:30.860612+0000","last_undegraded":"2024-04-28T12:50:30.860612+0000","last_fullsized":"2024-04-28T12:50:30.860612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:22:34.657665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117993+0000","last_change":"2024-04-28T12:48:34.556933+0000","last_active":"2024-04-28T12:50:31.117993+0000","last_peered":"2024-04-28T12:50:31.117993+0000","last_clean":"2024-04-28T12:50:31.117993+0000","last_became_active":"2024-04-28T12:48:34.556628+0000","last_became_peered":"2024-04-28T12:48:34.556628+0000","last_unstale":"2024-04-28T12:50:31.117993+0000","last_undegraded":"2024-04-28T12:50:31.117993+0000","last_fullsized":"2024-04-28T12:50:31.117993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:40:50.464723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.909008+0000","last_change":"2024-04-28T12:48:39.594808+0000","last_active":"2024-04-28T12:50:30.909008+0000","last_peered":"2024-04-28T12:50:30.909008+0000","last_clean":"2024-04-28T12:50:30.909008+0000","last_became_active":"2024-04-28T12:48:39.594333+0000","last_became_peered":"2024-04-28T12:48:39.594333+0000","last_unstale":"2024-04-28T12:50:30.909008+0000","last_undegraded":"2024-04-28T12:50:30.909008+0000","last_fullsized":"2024-04-28T12:50:30.909008+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:34:51.548490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.857973+0000","last_change":"2024-04-28T12:48:33.836878+0000","last_active":"2024-04-28T12:50:30.857973+0000","last_peered":"2024-04-28T12:50:30.857973+0000","last_clean":"2024-04-28T12:50:30.857973+0000","last_became_active":"2024-04-28T12:48:33.836536+0000","last_became_peered":"2024-04-28T12:48:33.836536+0000","last_unstale":"2024-04-28T12:50:30.857973+0000","last_undegraded":"2024-04-28T12:50:30.857973+0000","last_fullsized":"2024-04-28T12:50:30.857973+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:50:41.064854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.861399+0000","last_change":"2024-04-28T12:48:34.575213+0000","last_active":"2024-04-28T12:50:30.861399+0000","last_peered":"2024-04-28T12:50:30.861399+0000","last_clean":"2024-04-28T12:50:30.861399+0000","last_became_active":"2024-04-28T12:48:34.574892+0000","last_became_peered":"2024-04-28T12:48:34.574892+0000","last_unstale":"2024-04-28T12:50:30.861399+0000","last_undegraded":"2024-04-28T12:50:30.861399+0000","last_fullsized":"2024-04-28T12:50:30.861399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:27:54.144215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.909055+0000","last_change":"2024-04-28T12:48:39.912466+0000","last_active":"2024-04-28T12:50:30.909055+0000","last_peered":"2024-04-28T12:50:30.909055+0000","last_clean":"2024-04-28T12:50:30.909055+0000","last_became_active":"2024-04-28T12:48:39.911914+0000","last_became_peered":"2024-04-28T12:48:39.911914+0000","last_unstale":"2024-04-28T12:50:30.909055+0000","last_undegraded":"2024-04-28T12:50:30.909055+0000","last_fullsized":"2024-04-28T12:50:30.909055+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:26:35.341389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998242+0000","last_change":"2024-04-28T12:48:33.840024+0000","last_active":"2024-04-28T12:50:30.998242+0000","last_peered":"2024-04-28T12:50:30.998242+0000","last_clean":"2024-04-28T12:50:30.998242+0000","last_became_active":"2024-04-28T12:48:33.838608+0000","last_became_peered":"2024-04-28T12:48:33.838608+0000","last_unstale":"2024-04-28T12:50:30.998242+0000","last_undegraded":"2024-04-28T12:50:30.998242+0000","last_fullsized":"2024-04-28T12:50:30.998242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:31:28.060385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118258+0000","last_change":"2024-04-28T12:48:34.554685+0000","last_active":"2024-04-28T12:50:31.118258+0000","last_peered":"2024-04-28T12:50:31.118258+0000","last_clean":"2024-04-28T12:50:31.118258+0000","last_became_active":"2024-04-28T12:48:34.554305+0000","last_became_peered":"2024-04-28T12:48:34.554305+0000","last_unstale":"2024-04-28T12:50:31.118258+0000","last_undegraded":"2024-04-28T12:50:31.118258+0000","last_fullsized":"2024-04-28T12:50:31.118258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:19:53.698623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.271751+0000","last_change":"2024-04-28T12:48:39.605409+0000","last_active":"2024-04-28T12:50:30.271751+0000","last_peered":"2024-04-28T12:50:30.271751+0000","last_clean":"2024-04-28T12:50:30.271751+0000","last_became_active":"2024-04-28T12:48:39.604991+0000","last_became_peered":"2024-04-28T12:48:39.604991+0000","last_unstale":"2024-04-28T12:50:30.271751+0000","last_undegraded":"2024-04-28T12:50:30.271751+0000","last_fullsized":"2024-04-28T12:50:30.271751+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:37:44.481211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858374+0000","last_change":"2024-04-28T12:48:33.835286+0000","last_active":"2024-04-28T12:50:30.858374+0000","last_peered":"2024-04-28T12:50:30.858374+0000","last_clean":"2024-04-28T12:50:30.858374+0000","last_became_active":"2024-04-28T12:48:33.834729+0000","last_became_peered":"2024-04-28T12:48:33.834729+0000","last_unstale":"2024-04-28T12:50:30.858374+0000","last_undegraded":"2024-04-28T12:50:30.858374+0000","last_fullsized":"2024-04-28T12:50:30.858374+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:52:54.889021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118286+0000","last_change":"2024-04-28T12:48:34.547865+0000","last_active":"2024-04-28T12:50:31.118286+0000","last_peered":"2024-04-28T12:50:31.118286+0000","last_clean":"2024-04-28T12:50:31.118286+0000","last_became_active":"2024-04-28T12:48:34.547226+0000","last_became_peered":"2024-04-28T12:48:34.547226+0000","last_unstale":"2024-04-28T12:50:31.118286+0000","last_undegraded":"2024-04-28T12:50:31.118286+0000","last_fullsized":"2024-04-28T12:50:31.118286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:14:06.376908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.120203+0000","last_change":"2024-04-28T12:49:30.994601+0000","last_active":"2024-04-28T12:50:31.120203+0000","last_peered":"2024-04-28T12:50:31.120203+0000","last_clean":"2024-04-28T12:50:31.120203+0000","last_became_active":"2024-04-28T12:49:30.993830+0000","last_became_peered":"2024-04-28T12:49:30.993830+0000","last_unstale":"2024-04-28T12:50:31.120203+0000","last_undegraded":"2024-04-28T12:50:31.120203+0000","last_fullsized":"2024-04-28T12:50:31.120203+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T12:55:59.131768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.888203+0000","last_change":"2024-04-28T12:48:33.835109+0000","last_active":"2024-04-28T12:50:30.888203+0000","last_peered":"2024-04-28T12:50:30.888203+0000","last_clean":"2024-04-28T12:50:30.888203+0000","last_became_active":"2024-04-28T12:48:33.834757+0000","last_became_peered":"2024-04-28T12:48:33.834757+0000","last_unstale":"2024-04-28T12:50:30.888203+0000","last_undegraded":"2024-04-28T12:50:30.888203+0000","last_fullsized":"2024-04-28T12:50:30.888203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:46:45.884261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118322+0000","last_change":"2024-04-28T12:48:34.565135+0000","last_active":"2024-04-28T12:50:31.118322+0000","last_peered":"2024-04-28T12:50:31.118322+0000","last_clean":"2024-04-28T12:50:31.118322+0000","last_became_active":"2024-04-28T12:48:34.564631+0000","last_became_peered":"2024-04-28T12:48:34.564631+0000","last_unstale":"2024-04-28T12:50:31.118322+0000","last_undegraded":"2024-04-28T12:50:31.118322+0000","last_fullsized":"2024-04-28T12:50:31.118322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:36:46.747085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.855449+0000","last_change":"2024-04-28T12:49:30.993743+0000","last_active":"2024-04-28T12:50:30.855449+0000","last_peered":"2024-04-28T12:50:30.855449+0000","last_clean":"2024-04-28T12:50:30.855449+0000","last_became_active":"2024-04-28T12:49:30.993136+0000","last_became_peered":"2024-04-28T12:49:30.993136+0000","last_unstale":"2024-04-28T12:50:30.855449+0000","last_undegraded":"2024-04-28T12:50:30.855449+0000","last_fullsized":"2024-04-28T12:50:30.855449+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:53:32.903663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,7,1],"acting":[10,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.907114+0000","last_change":"2024-04-28T12:48:33.835565+0000","last_active":"2024-04-28T12:50:30.907114+0000","last_peered":"2024-04-28T12:50:30.907114+0000","last_clean":"2024-04-28T12:50:30.907114+0000","last_became_active":"2024-04-28T12:48:33.835147+0000","last_became_peered":"2024-04-28T12:48:33.835147+0000","last_unstale":"2024-04-28T12:50:30.907114+0000","last_undegraded":"2024-04-28T12:50:30.907114+0000","last_fullsized":"2024-04-28T12:50:30.907114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:48:23.520295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118359+0000","last_change":"2024-04-28T12:48:34.547561+0000","last_active":"2024-04-28T12:50:31.118359+0000","last_peered":"2024-04-28T12:50:31.118359+0000","last_clean":"2024-04-28T12:50:31.118359+0000","last_became_active":"2024-04-28T12:48:34.547236+0000","last_became_peered":"2024-04-28T12:48:34.547236+0000","last_unstale":"2024-04-28T12:50:31.118359+0000","last_undegraded":"2024-04-28T12:50:31.118359+0000","last_fullsized":"2024-04-28T12:50:31.118359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:00:37.150080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118019+0000","last_change":"2024-04-28T12:48:34.546027+0000","last_active":"2024-04-28T12:50:31.118019+0000","last_peered":"2024-04-28T12:50:31.118019+0000","last_clean":"2024-04-28T12:50:31.118019+0000","last_became_active":"2024-04-28T12:48:34.545618+0000","last_became_peered":"2024-04-28T12:48:34.545618+0000","last_unstale":"2024-04-28T12:50:31.118019+0000","last_undegraded":"2024-04-28T12:50:31.118019+0000","last_fullsized":"2024-04-28T12:50:31.118019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:40:00.928586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858151+0000","last_change":"2024-04-28T12:48:34.544883+0000","last_active":"2024-04-28T12:50:30.858151+0000","last_peered":"2024-04-28T12:50:30.858151+0000","last_clean":"2024-04-28T12:50:30.858151+0000","last_became_active":"2024-04-28T12:48:34.544454+0000","last_became_peered":"2024-04-28T12:48:34.544454+0000","last_unstale":"2024-04-28T12:50:30.858151+0000","last_undegraded":"2024-04-28T12:50:30.858151+0000","last_fullsized":"2024-04-28T12:50:30.858151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:58:16.064432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.857917+0000","last_change":"2024-04-28T12:48:34.565110+0000","last_active":"2024-04-28T12:50:30.857917+0000","last_peered":"2024-04-28T12:50:30.857917+0000","last_clean":"2024-04-28T12:50:30.857917+0000","last_became_active":"2024-04-28T12:48:34.564723+0000","last_became_peered":"2024-04-28T12:48:34.564723+0000","last_unstale":"2024-04-28T12:50:30.857917+0000","last_undegraded":"2024-04-28T12:50:30.857917+0000","last_fullsized":"2024-04-28T12:50:30.857917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:58:24.220515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.121000+0000","last_change":"2024-04-28T12:48:34.572428+0000","last_active":"2024-04-28T12:50:31.121000+0000","last_peered":"2024-04-28T12:50:31.121000+0000","last_clean":"2024-04-28T12:50:31.121000+0000","last_became_active":"2024-04-28T12:48:34.572034+0000","last_became_peered":"2024-04-28T12:48:34.572034+0000","last_unstale":"2024-04-28T12:50:31.121000+0000","last_undegraded":"2024-04-28T12:50:31.121000+0000","last_fullsized":"2024-04-28T12:50:31.121000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:38:23.598568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.906819+0000","last_change":"2024-04-28T12:48:34.562425+0000","last_active":"2024-04-28T12:50:30.906819+0000","last_peered":"2024-04-28T12:50:30.906819+0000","last_clean":"2024-04-28T12:50:30.906819+0000","last_became_active":"2024-04-28T12:48:34.562087+0000","last_became_peered":"2024-04-28T12:48:34.562087+0000","last_unstale":"2024-04-28T12:50:30.906819+0000","last_undegraded":"2024-04-28T12:50:30.906819+0000","last_fullsized":"2024-04-28T12:50:30.906819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:32:39.701796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117894+0000","last_change":"2024-04-28T12:48:34.547755+0000","last_active":"2024-04-28T12:50:31.117894+0000","last_peered":"2024-04-28T12:50:31.117894+0000","last_clean":"2024-04-28T12:50:31.117894+0000","last_became_active":"2024-04-28T12:48:34.547427+0000","last_became_peered":"2024-04-28T12:48:34.547427+0000","last_unstale":"2024-04-28T12:50:31.117894+0000","last_undegraded":"2024-04-28T12:50:31.117894+0000","last_fullsized":"2024-04-28T12:50:31.117894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:58:15.694252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858531+0000","last_change":"2024-04-28T12:48:34.543146+0000","last_active":"2024-04-28T12:50:30.858531+0000","last_peered":"2024-04-28T12:50:30.858531+0000","last_clean":"2024-04-28T12:50:30.858531+0000","last_became_active":"2024-04-28T12:48:34.542809+0000","last_became_peered":"2024-04-28T12:48:34.542809+0000","last_unstale":"2024-04-28T12:50:30.858531+0000","last_undegraded":"2024-04-28T12:50:30.858531+0000","last_fullsized":"2024-04-28T12:50:30.858531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:09:42.515358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.863968+0000","last_change":"2024-04-28T12:48:34.571166+0000","last_active":"2024-04-28T12:50:30.863968+0000","last_peered":"2024-04-28T12:50:30.863968+0000","last_clean":"2024-04-28T12:50:30.863968+0000","last_became_active":"2024-04-28T12:48:34.567803+0000","last_became_peered":"2024-04-28T12:48:34.567803+0000","last_unstale":"2024-04-28T12:50:30.863968+0000","last_undegraded":"2024-04-28T12:50:30.863968+0000","last_fullsized":"2024-04-28T12:50:30.863968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:11:09.945751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887491+0000","last_change":"2024-04-28T12:48:34.564939+0000","last_active":"2024-04-28T12:50:30.887491+0000","last_peered":"2024-04-28T12:50:30.887491+0000","last_clean":"2024-04-28T12:50:30.887491+0000","last_became_active":"2024-04-28T12:48:34.564486+0000","last_became_peered":"2024-04-28T12:48:34.564486+0000","last_unstale":"2024-04-28T12:50:30.887491+0000","last_undegraded":"2024-04-28T12:50:30.887491+0000","last_fullsized":"2024-04-28T12:50:30.887491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:29:44.618469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.860774+0000","last_change":"2024-04-28T12:48:34.567359+0000","last_active":"2024-04-28T12:50:30.860774+0000","last_peered":"2024-04-28T12:50:30.860774+0000","last_clean":"2024-04-28T12:50:30.860774+0000","last_became_active":"2024-04-28T12:48:34.565393+0000","last_became_peered":"2024-04-28T12:48:34.565393+0000","last_unstale":"2024-04-28T12:50:30.860774+0000","last_undegraded":"2024-04-28T12:50:30.860774+0000","last_fullsized":"2024-04-28T12:50:30.860774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:44:04.312257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.272030+0000","last_change":"2024-04-28T12:48:34.565019+0000","last_active":"2024-04-28T12:50:30.272030+0000","last_peered":"2024-04-28T12:50:30.272030+0000","last_clean":"2024-04-28T12:50:30.272030+0000","last_became_active":"2024-04-28T12:48:34.564635+0000","last_became_peered":"2024-04-28T12:48:34.564635+0000","last_unstale":"2024-04-28T12:50:30.272030+0000","last_undegraded":"2024-04-28T12:50:30.272030+0000","last_fullsized":"2024-04-28T12:50:30.272030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:20.645281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.906591+0000","last_change":"2024-04-28T12:48:34.574043+0000","last_active":"2024-04-28T12:50:30.906591+0000","last_peered":"2024-04-28T12:50:30.906591+0000","last_clean":"2024-04-28T12:50:30.906591+0000","last_became_active":"2024-04-28T12:48:34.573686+0000","last_became_peered":"2024-04-28T12:48:34.573686+0000","last_unstale":"2024-04-28T12:50:30.906591+0000","last_undegraded":"2024-04-28T12:50:30.906591+0000","last_fullsized":"2024-04-28T12:50:30.906591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:29:57.371318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117689+0000","last_change":"2024-04-28T12:49:30.710555+0000","last_active":"2024-04-28T12:50:31.117689+0000","last_peered":"2024-04-28T12:50:31.117689+0000","last_clean":"2024-04-28T12:50:31.117689+0000","last_became_active":"2024-04-28T12:49:30.710205+0000","last_became_peered":"2024-04-28T12:49:30.710205+0000","last_unstale":"2024-04-28T12:50:31.117689+0000","last_undegraded":"2024-04-28T12:50:31.117689+0000","last_fullsized":"2024-04-28T12:50:31.117689+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:40:53.390140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281028,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59218,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":546,"num_read_kb":336,"num_write":164,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":9,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":177654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":527681,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1171456,"data_stored":1170166,"data_compressed":62465,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":532013,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070519,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1048576,"data_stored":527837,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1085440,"data_stored":1114808,"data_compressed":59457,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":523093,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1077248,"data_stored":568500,"data_compressed":23712,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":524481,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1114112,"data_stored":579570,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":523287,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":528134,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1105920,"data_stored":1119334,"data_compressed":62465,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34101,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34101,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34101,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":56567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":56795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-28T12:50:37.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-28T12:50:37.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:37 smithi084 ceph-mon[28081]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:50:37.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:37 smithi084 ceph-mon[28081]: pgmap v388: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 7 op/s 2024-04-28T12:50:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:37 smithi161 ceph-mon[62387]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:50:37.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:37 smithi161 ceph-mon[62387]: pgmap v388: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 7 op/s 2024-04-28T12:50:38.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:37 smithi179 ceph-mon[63029]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:50:38.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:37 smithi179 ceph-mon[63029]: pgmap v388: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 7 op/s 2024-04-28T12:50:38.180 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:39.909 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:39 smithi084 ceph-mon[28081]: pgmap v389: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 104 B/s wr, 4 op/s 2024-04-28T12:50:39.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:39 smithi161 ceph-mon[62387]: pgmap v389: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 104 B/s wr, 4 op/s 2024-04-28T12:50:40.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:39 smithi179 ceph-mon[63029]: pgmap v389: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 104 B/s wr, 4 op/s 2024-04-28T12:50:40.448 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.451+0000 7f3928800700 1 -- 172.21.15.84:0/1365409071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3920103470 msgr2=0x7f39201038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:40.448 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.451+0000 7f3928800700 1 --2- 172.21.15.84:0/1365409071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3920103470 0x7f39201038f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3910009990 tx=0x7f3910009ca0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.448 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.451+0000 7f3928800700 1 -- 172.21.15.84:0/1365409071 shutdown_connections 2024-04-28T12:50:40.448 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.451+0000 7f3928800700 1 --2- 172.21.15.84:0/1365409071 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39201094c0 0x7f39201098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.449 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.451+0000 7f3928800700 1 --2- 172.21.15.84:0/1365409071 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3920103e30 0x7f3920110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.449 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.451+0000 7f3928800700 1 --2- 172.21.15.84:0/1365409071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3920103470 0x7f39201038f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.449 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.451+0000 7f3928800700 1 -- 172.21.15.84:0/1365409071 >> 172.21.15.84:0/1365409071 conn(0x7f392007a580 msgr2=0x7f392007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:40.449 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.451+0000 7f3928800700 1 -- 172.21.15.84:0/1365409071 shutdown_connections 2024-04-28T12:50:40.449 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.451+0000 7f3928800700 1 -- 172.21.15.84:0/1365409071 wait complete. 2024-04-28T12:50:40.449 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.452+0000 7f3928800700 1 Processor -- start 2024-04-28T12:50:40.449 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.452+0000 7f3928800700 1 -- start start 2024-04-28T12:50:40.451 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.452+0000 7f3928800700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3920103470 0x7f3920075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:40.451 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.452+0000 7f3928800700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3920103e30 0x7f3920075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:40.451 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3928800700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39201094c0 0x7f39200761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:40.451 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3928800700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f39201135f0 con 0x7f3920103470 2024-04-28T12:50:40.451 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3928800700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3920113470 con 0x7f3920103e30 2024-04-28T12:50:40.451 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3928800700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3920113770 con 0x7f39201094c0 2024-04-28T12:50:40.451 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3925e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3920103470 0x7f3920075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:40.451 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3925400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3920103e30 0x7f3920075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:40.451 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3925e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3920103470 0x7f3920075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55426/0 (socket says 172.21.15.84:55426) 2024-04-28T12:50:40.451 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3925e00700 1 -- 172.21.15.84:0/4116421054 learned_addr learned my addr 172.21.15.84:0/4116421054 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3926800700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39201094c0 0x7f39200761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3925e00700 1 -- 172.21.15.84:0/4116421054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39201094c0 msgr2=0x7f39200761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3925e00700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39201094c0 0x7f39200761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3925e00700 1 -- 172.21.15.84:0/4116421054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3920103e30 msgr2=0x7f3920075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3925e00700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3920103e30 0x7f3920075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3925e00700 1 -- 172.21.15.84:0/4116421054 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39201a8050 con 0x7f3920103470 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.453+0000 7f3925400700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3920103e30 0x7f3920075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.454+0000 7f3925e00700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3920103470 0x7f3920075720 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f391400d980 tx=0x7f391400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.454+0000 7f391aa00700 1 -- 172.21.15.84:0/4116421054 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914016070 con 0x7f3920103470 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.454+0000 7f391aa00700 1 -- 172.21.15.84:0/4116421054 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3914004830 con 0x7f3920103470 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.454+0000 7f3928800700 1 -- 172.21.15.84:0/4116421054 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39201a8340 con 0x7f3920103470 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.454+0000 7f391aa00700 1 -- 172.21.15.84:0/4116421054 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914005540 con 0x7f3920103470 2024-04-28T12:50:40.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.454+0000 7f3928800700 1 -- 172.21.15.84:0/4116421054 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39201a88a0 con 0x7f3920103470 2024-04-28T12:50:40.453 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.456+0000 7f3928800700 1 -- 172.21.15.84:0/4116421054 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f392010b930 con 0x7f3920103470 2024-04-28T12:50:40.454 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.458+0000 7f391aa00700 1 -- 172.21.15.84:0/4116421054 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3914004220 con 0x7f3920103470 2024-04-28T12:50:40.455 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.459+0000 7f391aa00700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f39080a3d50 0x7f39080a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:40.455 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.459+0000 7f391aa00700 1 -- 172.21.15.84:0/4116421054 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f39140d05e0 con 0x7f3920103470 2024-04-28T12:50:40.455 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.459+0000 7f3925400700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f39080a3d50 0x7f39080a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:40.456 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.460+0000 7f3925400700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f39080a3d50 0x7f39080a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f391000b2e0 tx=0x7f391001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:40.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.464+0000 7f391aa00700 1 -- 172.21.15.84:0/4116421054 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f391401f020 con 0x7f3920103470 2024-04-28T12:50:40.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.696+0000 7f3928800700 1 -- 172.21.15.84:0/4116421054 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3920063860 con 0x7f39080a3d50 2024-04-28T12:50:40.714 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.718+0000 7f391aa00700 1 -- 172.21.15.84:0/4116421054 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340678 (secure 0 0 0) 0x7f3920063860 con 0x7f39080a3d50 2024-04-28T12:50:40.721 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:50:40.723 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.725+0000 7f3928800700 1 -- 172.21.15.84:0/4116421054 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f39080a3d50 msgr2=0x7f39080a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:40.723 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.725+0000 7f3928800700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f39080a3d50 0x7f39080a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f391000b2e0 tx=0x7f391001a040 comp rx=0 tx=0).stop 2024-04-28T12:50:40.723 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.725+0000 7f3928800700 1 -- 172.21.15.84:0/4116421054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3920103470 msgr2=0x7f3920075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:40.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.725+0000 7f3928800700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3920103470 0x7f3920075720 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f391400d980 tx=0x7f391400dc90 comp rx=0 tx=0).stop 2024-04-28T12:50:40.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.725+0000 7f3928800700 1 -- 172.21.15.84:0/4116421054 shutdown_connections 2024-04-28T12:50:40.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.725+0000 7f3928800700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39201094c0 0x7f39200761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.725+0000 7f3928800700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3920103470 0x7f3920075720 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.725+0000 7f3928800700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f39080a3d50 0x7f39080a6210 secure :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f391000b2e0 tx=0x7f391001a040 comp rx=0 tx=0).stop 2024-04-28T12:50:40.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.725+0000 7f3928800700 1 --2- 172.21.15.84:0/4116421054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3920103e30 0x7f3920075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:40.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.725+0000 7f3928800700 1 -- 172.21.15.84:0/4116421054 >> 172.21.15.84:0/4116421054 conn(0x7f392007a580 msgr2=0x7f39201022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:40.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.726+0000 7f3928800700 1 -- 172.21.15.84:0/4116421054 shutdown_connections 2024-04-28T12:50:40.724 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:40.726+0000 7f3928800700 1 -- 172.21.15.84:0/4116421054 wait complete. 2024-04-28T12:50:40.725 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-04-28T12:50:41.516 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-04-28T12:50:39.655372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649820,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":672,"num_read_kb":445,"num_write":347,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":9,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365956,"kb_used_data":12572,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575868,"statfs":{"total":1151940427776,"available":1151565688832,"internally_reserved":0,"allocated":12873728,"data_stored":8257562,"data_compressed":427875,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":20,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.803993"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118023+0000","last_change":"2024-04-28T12:48:34.569210+0000","last_active":"2024-04-28T12:50:31.118023+0000","last_peered":"2024-04-28T12:50:31.118023+0000","last_clean":"2024-04-28T12:50:31.118023+0000","last_became_active":"2024-04-28T12:48:34.568912+0000","last_became_peered":"2024-04-28T12:48:34.568912+0000","last_unstale":"2024-04-28T12:50:31.118023+0000","last_undegraded":"2024-04-28T12:50:31.118023+0000","last_fullsized":"2024-04-28T12:50:31.118023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:48:26.513370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.120476+0000","last_change":"2024-04-28T12:48:34.577664+0000","last_active":"2024-04-28T12:50:31.120476+0000","last_peered":"2024-04-28T12:50:31.120476+0000","last_clean":"2024-04-28T12:50:31.120476+0000","last_became_active":"2024-04-28T12:48:34.576679+0000","last_became_peered":"2024-04-28T12:48:34.576679+0000","last_unstale":"2024-04-28T12:50:31.120476+0000","last_undegraded":"2024-04-28T12:50:31.120476+0000","last_fullsized":"2024-04-28T12:50:31.120476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:26:44.807081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.119777+0000","last_change":"2024-04-28T12:48:34.573666+0000","last_active":"2024-04-28T12:50:31.119777+0000","last_peered":"2024-04-28T12:50:31.119777+0000","last_clean":"2024-04-28T12:50:31.119777+0000","last_became_active":"2024-04-28T12:48:34.573111+0000","last_became_peered":"2024-04-28T12:48:34.573111+0000","last_unstale":"2024-04-28T12:50:31.119777+0000","last_undegraded":"2024-04-28T12:50:31.119777+0000","last_fullsized":"2024-04-28T12:50:31.119777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:41:13.586589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.909372+0000","last_change":"2024-04-28T12:48:34.543140+0000","last_active":"2024-04-28T12:50:30.909372+0000","last_peered":"2024-04-28T12:50:30.909372+0000","last_clean":"2024-04-28T12:50:30.909372+0000","last_became_active":"2024-04-28T12:48:34.542803+0000","last_became_peered":"2024-04-28T12:48:34.542803+0000","last_unstale":"2024-04-28T12:50:30.909372+0000","last_undegraded":"2024-04-28T12:50:30.909372+0000","last_fullsized":"2024-04-28T12:50:30.909372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:46:47.313190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998318+0000","last_change":"2024-04-28T12:48:34.559988+0000","last_active":"2024-04-28T12:50:30.998318+0000","last_peered":"2024-04-28T12:50:30.998318+0000","last_clean":"2024-04-28T12:50:30.998318+0000","last_became_active":"2024-04-28T12:48:34.559657+0000","last_became_peered":"2024-04-28T12:48:34.559657+0000","last_unstale":"2024-04-28T12:50:30.998318+0000","last_undegraded":"2024-04-28T12:50:30.998318+0000","last_fullsized":"2024-04-28T12:50:30.998318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:13:27.610212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.855834+0000","last_change":"2024-04-28T12:48:34.553539+0000","last_active":"2024-04-28T12:50:30.855834+0000","last_peered":"2024-04-28T12:50:30.855834+0000","last_clean":"2024-04-28T12:50:30.855834+0000","last_became_active":"2024-04-28T12:48:34.553017+0000","last_became_peered":"2024-04-28T12:48:34.553017+0000","last_unstale":"2024-04-28T12:50:30.855834+0000","last_undegraded":"2024-04-28T12:50:30.855834+0000","last_fullsized":"2024-04-28T12:50:30.855834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:31:14.895894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117554+0000","last_change":"2024-04-28T12:48:34.553305+0000","last_active":"2024-04-28T12:50:31.117554+0000","last_peered":"2024-04-28T12:50:31.117554+0000","last_clean":"2024-04-28T12:50:31.117554+0000","last_became_active":"2024-04-28T12:48:34.552950+0000","last_became_peered":"2024-04-28T12:48:34.552950+0000","last_unstale":"2024-04-28T12:50:31.117554+0000","last_undegraded":"2024-04-28T12:50:31.117554+0000","last_fullsized":"2024-04-28T12:50:31.117554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:01:43.903986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.859871+0000","last_change":"2024-04-28T12:48:34.558899+0000","last_active":"2024-04-28T12:50:30.859871+0000","last_peered":"2024-04-28T12:50:30.859871+0000","last_clean":"2024-04-28T12:50:30.859871+0000","last_became_active":"2024-04-28T12:48:34.558507+0000","last_became_peered":"2024-04-28T12:48:34.558507+0000","last_unstale":"2024-04-28T12:50:30.859871+0000","last_undegraded":"2024-04-28T12:50:30.859871+0000","last_fullsized":"2024-04-28T12:50:30.859871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:46:28.059231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117703+0000","last_change":"2024-04-28T12:48:34.545488+0000","last_active":"2024-04-28T12:50:31.117703+0000","last_peered":"2024-04-28T12:50:31.117703+0000","last_clean":"2024-04-28T12:50:31.117703+0000","last_became_active":"2024-04-28T12:48:34.545168+0000","last_became_peered":"2024-04-28T12:48:34.545168+0000","last_unstale":"2024-04-28T12:50:31.117703+0000","last_undegraded":"2024-04-28T12:50:31.117703+0000","last_fullsized":"2024-04-28T12:50:31.117703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:18:02.174760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.856319+0000","last_change":"2024-04-28T12:48:34.544213+0000","last_active":"2024-04-28T12:50:30.856319+0000","last_peered":"2024-04-28T12:50:30.856319+0000","last_clean":"2024-04-28T12:50:30.856319+0000","last_became_active":"2024-04-28T12:48:34.543910+0000","last_became_peered":"2024-04-28T12:48:34.543910+0000","last_unstale":"2024-04-28T12:50:30.856319+0000","last_undegraded":"2024-04-28T12:50:30.856319+0000","last_fullsized":"2024-04-28T12:50:30.856319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:09:52.602695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118151+0000","last_change":"2024-04-28T12:48:34.548121+0000","last_active":"2024-04-28T12:50:31.118151+0000","last_peered":"2024-04-28T12:50:31.118151+0000","last_clean":"2024-04-28T12:50:31.118151+0000","last_became_active":"2024-04-28T12:48:34.547822+0000","last_became_peered":"2024-04-28T12:48:34.547822+0000","last_unstale":"2024-04-28T12:50:31.118151+0000","last_undegraded":"2024-04-28T12:50:31.118151+0000","last_fullsized":"2024-04-28T12:50:31.118151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T12:49:39.838959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.909308+0000","last_change":"2024-04-28T12:48:34.544146+0000","last_active":"2024-04-28T12:50:30.909308+0000","last_peered":"2024-04-28T12:50:30.909308+0000","last_clean":"2024-04-28T12:50:30.909308+0000","last_became_active":"2024-04-28T12:48:34.543805+0000","last_became_peered":"2024-04-28T12:48:34.543805+0000","last_unstale":"2024-04-28T12:50:30.909308+0000","last_undegraded":"2024-04-28T12:50:30.909308+0000","last_fullsized":"2024-04-28T12:50:30.909308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:06:46.871316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.906406+0000","last_change":"2024-04-28T12:48:34.536765+0000","last_active":"2024-04-28T12:50:30.906406+0000","last_peered":"2024-04-28T12:50:30.906406+0000","last_clean":"2024-04-28T12:50:30.906406+0000","last_became_active":"2024-04-28T12:48:34.535398+0000","last_became_peered":"2024-04-28T12:48:34.535398+0000","last_unstale":"2024-04-28T12:50:30.906406+0000","last_undegraded":"2024-04-28T12:50:30.906406+0000","last_fullsized":"2024-04-28T12:50:30.906406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:54:25.100414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.119606+0000","last_change":"2024-04-28T12:48:34.573782+0000","last_active":"2024-04-28T12:50:31.119606+0000","last_peered":"2024-04-28T12:50:31.119606+0000","last_clean":"2024-04-28T12:50:31.119606+0000","last_became_active":"2024-04-28T12:48:34.573361+0000","last_became_peered":"2024-04-28T12:48:34.573361+0000","last_unstale":"2024-04-28T12:50:31.119606+0000","last_undegraded":"2024-04-28T12:50:31.119606+0000","last_fullsized":"2024-04-28T12:50:31.119606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:29:02.746038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.119979+0000","last_change":"2024-04-28T12:48:34.570810+0000","last_active":"2024-04-28T12:50:31.119979+0000","last_peered":"2024-04-28T12:50:31.119979+0000","last_clean":"2024-04-28T12:50:31.119979+0000","last_became_active":"2024-04-28T12:48:34.570337+0000","last_became_peered":"2024-04-28T12:48:34.570337+0000","last_unstale":"2024-04-28T12:50:31.119979+0000","last_undegraded":"2024-04-28T12:50:31.119979+0000","last_fullsized":"2024-04-28T12:50:31.119979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:41:36.028710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117408+0000","last_change":"2024-04-28T12:48:34.558802+0000","last_active":"2024-04-28T12:50:31.117408+0000","last_peered":"2024-04-28T12:50:31.117408+0000","last_clean":"2024-04-28T12:50:31.117408+0000","last_became_active":"2024-04-28T12:48:34.558425+0000","last_became_peered":"2024-04-28T12:48:34.558425+0000","last_unstale":"2024-04-28T12:50:31.117408+0000","last_undegraded":"2024-04-28T12:50:31.117408+0000","last_fullsized":"2024-04-28T12:50:31.117408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:03:22.681965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117984+0000","last_change":"2024-04-28T12:48:34.569546+0000","last_active":"2024-04-28T12:50:31.117984+0000","last_peered":"2024-04-28T12:50:31.117984+0000","last_clean":"2024-04-28T12:50:31.117984+0000","last_became_active":"2024-04-28T12:48:34.569231+0000","last_became_peered":"2024-04-28T12:48:34.569231+0000","last_unstale":"2024-04-28T12:50:31.117984+0000","last_undegraded":"2024-04-28T12:50:31.117984+0000","last_fullsized":"2024-04-28T12:50:31.117984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:19:24.491204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.869122+0000","last_change":"2024-04-28T12:48:34.555634+0000","last_active":"2024-04-28T12:50:30.869122+0000","last_peered":"2024-04-28T12:50:30.869122+0000","last_clean":"2024-04-28T12:50:30.869122+0000","last_became_active":"2024-04-28T12:48:34.555054+0000","last_became_peered":"2024-04-28T12:48:34.555054+0000","last_unstale":"2024-04-28T12:50:30.869122+0000","last_undegraded":"2024-04-28T12:50:30.869122+0000","last_fullsized":"2024-04-28T12:50:30.869122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:36:37.493994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858456+0000","last_change":"2024-04-28T12:48:34.568857+0000","last_active":"2024-04-28T12:50:30.858456+0000","last_peered":"2024-04-28T12:50:30.858456+0000","last_clean":"2024-04-28T12:50:30.858456+0000","last_became_active":"2024-04-28T12:48:34.568550+0000","last_became_peered":"2024-04-28T12:48:34.568550+0000","last_unstale":"2024-04-28T12:50:30.858456+0000","last_undegraded":"2024-04-28T12:50:30.858456+0000","last_fullsized":"2024-04-28T12:50:30.858456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:09:40.709446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858104+0000","last_change":"2024-04-28T12:48:39.608021+0000","last_active":"2024-04-28T12:50:30.858104+0000","last_peered":"2024-04-28T12:50:30.858104+0000","last_clean":"2024-04-28T12:50:30.858104+0000","last_became_active":"2024-04-28T12:48:39.607187+0000","last_became_peered":"2024-04-28T12:48:39.607187+0000","last_unstale":"2024-04-28T12:50:30.858104+0000","last_undegraded":"2024-04-28T12:50:30.858104+0000","last_fullsized":"2024-04-28T12:50:30.858104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:31:26.346414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.869351+0000","last_change":"2024-04-28T12:48:33.839179+0000","last_active":"2024-04-28T12:50:30.869351+0000","last_peered":"2024-04-28T12:50:30.869351+0000","last_clean":"2024-04-28T12:50:30.869351+0000","last_became_active":"2024-04-28T12:48:33.838004+0000","last_became_peered":"2024-04-28T12:48:33.838004+0000","last_unstale":"2024-04-28T12:50:30.869351+0000","last_undegraded":"2024-04-28T12:50:30.869351+0000","last_fullsized":"2024-04-28T12:50:30.869351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:32:14.765652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.861507+0000","last_change":"2024-04-28T12:48:34.571840+0000","last_active":"2024-04-28T12:50:30.861507+0000","last_peered":"2024-04-28T12:50:30.861507+0000","last_clean":"2024-04-28T12:50:30.861507+0000","last_became_active":"2024-04-28T12:48:34.568001+0000","last_became_peered":"2024-04-28T12:48:34.568001+0000","last_unstale":"2024-04-28T12:50:30.861507+0000","last_undegraded":"2024-04-28T12:50:30.861507+0000","last_fullsized":"2024-04-28T12:50:30.861507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:15:22.276123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.908708+0000","last_change":"2024-04-28T12:48:39.605055+0000","last_active":"2024-04-28T12:50:30.908708+0000","last_peered":"2024-04-28T12:50:30.908708+0000","last_clean":"2024-04-28T12:50:30.908708+0000","last_became_active":"2024-04-28T12:48:39.604566+0000","last_became_peered":"2024-04-28T12:48:39.604566+0000","last_unstale":"2024-04-28T12:50:30.908708+0000","last_undegraded":"2024-04-28T12:50:30.908708+0000","last_fullsized":"2024-04-28T12:50:30.908708+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:30:48.945467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.862441+0000","last_change":"2024-04-28T12:48:33.841745+0000","last_active":"2024-04-28T12:50:30.862441+0000","last_peered":"2024-04-28T12:50:30.862441+0000","last_clean":"2024-04-28T12:50:30.862441+0000","last_became_active":"2024-04-28T12:48:33.835003+0000","last_became_peered":"2024-04-28T12:48:33.835003+0000","last_unstale":"2024-04-28T12:50:30.862441+0000","last_undegraded":"2024-04-28T12:50:30.862441+0000","last_fullsized":"2024-04-28T12:50:30.862441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:40:14.253322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.120711+0000","last_change":"2024-04-28T12:48:34.577688+0000","last_active":"2024-04-28T12:50:31.120711+0000","last_peered":"2024-04-28T12:50:31.120711+0000","last_clean":"2024-04-28T12:50:31.120711+0000","last_became_active":"2024-04-28T12:48:34.577269+0000","last_became_peered":"2024-04-28T12:48:34.577269+0000","last_unstale":"2024-04-28T12:50:31.120711+0000","last_undegraded":"2024-04-28T12:50:31.120711+0000","last_fullsized":"2024-04-28T12:50:31.120711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:30:13.744015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.908823+0000","last_change":"2024-04-28T12:48:39.600708+0000","last_active":"2024-04-28T12:50:30.908823+0000","last_peered":"2024-04-28T12:50:30.908823+0000","last_clean":"2024-04-28T12:50:30.908823+0000","last_became_active":"2024-04-28T12:48:39.600257+0000","last_became_peered":"2024-04-28T12:48:39.600257+0000","last_unstale":"2024-04-28T12:50:30.908823+0000","last_undegraded":"2024-04-28T12:50:30.908823+0000","last_fullsized":"2024-04-28T12:50:30.908823+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:08:02.685278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998632+0000","last_change":"2024-04-28T12:48:33.837824+0000","last_active":"2024-04-28T12:50:30.998632+0000","last_peered":"2024-04-28T12:50:30.998632+0000","last_clean":"2024-04-28T12:50:30.998632+0000","last_became_active":"2024-04-28T12:48:33.837505+0000","last_became_peered":"2024-04-28T12:48:33.837505+0000","last_unstale":"2024-04-28T12:50:30.998632+0000","last_undegraded":"2024-04-28T12:50:30.998632+0000","last_fullsized":"2024-04-28T12:50:30.998632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:21:10.677381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117739+0000","last_change":"2024-04-28T12:48:34.554575+0000","last_active":"2024-04-28T12:50:31.117739+0000","last_peered":"2024-04-28T12:50:31.117739+0000","last_clean":"2024-04-28T12:50:31.117739+0000","last_became_active":"2024-04-28T12:48:34.554214+0000","last_became_peered":"2024-04-28T12:48:34.554214+0000","last_unstale":"2024-04-28T12:50:31.117739+0000","last_undegraded":"2024-04-28T12:50:31.117739+0000","last_fullsized":"2024-04-28T12:50:31.117739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:37:03.369874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.908769+0000","last_change":"2024-04-28T12:48:39.604793+0000","last_active":"2024-04-28T12:50:30.908769+0000","last_peered":"2024-04-28T12:50:30.908769+0000","last_clean":"2024-04-28T12:50:30.908769+0000","last_became_active":"2024-04-28T12:48:39.604338+0000","last_became_peered":"2024-04-28T12:48:39.604338+0000","last_unstale":"2024-04-28T12:50:30.908769+0000","last_undegraded":"2024-04-28T12:50:30.908769+0000","last_fullsized":"2024-04-28T12:50:30.908769+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:14:56.225098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998310+0000","last_change":"2024-04-28T12:48:33.839375+0000","last_active":"2024-04-28T12:50:30.998310+0000","last_peered":"2024-04-28T12:50:30.998310+0000","last_clean":"2024-04-28T12:50:30.998310+0000","last_became_active":"2024-04-28T12:48:33.839062+0000","last_became_peered":"2024-04-28T12:48:33.839062+0000","last_unstale":"2024-04-28T12:50:30.998310+0000","last_undegraded":"2024-04-28T12:50:30.998310+0000","last_fullsized":"2024-04-28T12:50:30.998310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:20:06.692743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.888038+0000","last_change":"2024-04-28T12:48:34.543341+0000","last_active":"2024-04-28T12:50:30.888038+0000","last_peered":"2024-04-28T12:50:30.888038+0000","last_clean":"2024-04-28T12:50:30.888038+0000","last_became_active":"2024-04-28T12:48:34.543033+0000","last_became_peered":"2024-04-28T12:48:34.543033+0000","last_unstale":"2024-04-28T12:50:30.888038+0000","last_undegraded":"2024-04-28T12:50:30.888038+0000","last_fullsized":"2024-04-28T12:50:30.888038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:41:27.654577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858212+0000","last_change":"2024-04-28T12:48:39.589856+0000","last_active":"2024-04-28T12:50:30.858212+0000","last_peered":"2024-04-28T12:50:30.858212+0000","last_clean":"2024-04-28T12:50:30.858212+0000","last_became_active":"2024-04-28T12:48:39.589148+0000","last_became_peered":"2024-04-28T12:48:39.589148+0000","last_unstale":"2024-04-28T12:50:30.858212+0000","last_undegraded":"2024-04-28T12:50:30.858212+0000","last_fullsized":"2024-04-28T12:50:30.858212+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:34:11.897109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":128,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:32.891531+0000","last_change":"2024-04-28T12:48:33.829387+0000","last_active":"2024-04-28T12:50:32.891531+0000","last_peered":"2024-04-28T12:50:32.891531+0000","last_clean":"2024-04-28T12:50:32.891531+0000","last_became_active":"2024-04-28T12:48:33.829011+0000","last_became_peered":"2024-04-28T12:48:33.829011+0000","last_unstale":"2024-04-28T12:50:32.891531+0000","last_undegraded":"2024-04-28T12:50:32.891531+0000","last_fullsized":"2024-04-28T12:50:32.891531+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:54:22.096246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998139+0000","last_change":"2024-04-28T12:48:34.554629+0000","last_active":"2024-04-28T12:50:30.998139+0000","last_peered":"2024-04-28T12:50:30.998139+0000","last_clean":"2024-04-28T12:50:30.998139+0000","last_became_active":"2024-04-28T12:48:34.554231+0000","last_became_peered":"2024-04-28T12:48:34.554231+0000","last_unstale":"2024-04-28T12:50:30.998139+0000","last_undegraded":"2024-04-28T12:50:30.998139+0000","last_fullsized":"2024-04-28T12:50:30.998139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:21:53.706120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858211+0000","last_change":"2024-04-28T12:48:39.606416+0000","last_active":"2024-04-28T12:50:30.858211+0000","last_peered":"2024-04-28T12:50:30.858211+0000","last_clean":"2024-04-28T12:50:30.858211+0000","last_became_active":"2024-04-28T12:48:39.605957+0000","last_became_peered":"2024-04-28T12:48:39.605957+0000","last_unstale":"2024-04-28T12:50:30.858211+0000","last_undegraded":"2024-04-28T12:50:30.858211+0000","last_fullsized":"2024-04-28T12:50:30.858211+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:39:16.315446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.855726+0000","last_change":"2024-04-28T12:48:33.841119+0000","last_active":"2024-04-28T12:50:30.855726+0000","last_peered":"2024-04-28T12:50:30.855726+0000","last_clean":"2024-04-28T12:50:30.855726+0000","last_became_active":"2024-04-28T12:48:33.840833+0000","last_became_peered":"2024-04-28T12:48:33.840833+0000","last_unstale":"2024-04-28T12:50:30.855726+0000","last_undegraded":"2024-04-28T12:50:30.855726+0000","last_fullsized":"2024-04-28T12:50:30.855726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:49:31.145968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887846+0000","last_change":"2024-04-28T12:48:34.565140+0000","last_active":"2024-04-28T12:50:30.887846+0000","last_peered":"2024-04-28T12:50:30.887846+0000","last_clean":"2024-04-28T12:50:30.887846+0000","last_became_active":"2024-04-28T12:48:34.564692+0000","last_became_peered":"2024-04-28T12:48:34.564692+0000","last_unstale":"2024-04-28T12:50:30.887846+0000","last_undegraded":"2024-04-28T12:50:30.887846+0000","last_fullsized":"2024-04-28T12:50:30.887846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:55:27.440482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117850+0000","last_change":"2024-04-28T12:48:39.611109+0000","last_active":"2024-04-28T12:50:31.117850+0000","last_peered":"2024-04-28T12:50:31.117850+0000","last_clean":"2024-04-28T12:50:31.117850+0000","last_became_active":"2024-04-28T12:48:39.610754+0000","last_became_peered":"2024-04-28T12:48:39.610754+0000","last_unstale":"2024-04-28T12:50:31.117850+0000","last_undegraded":"2024-04-28T12:50:31.117850+0000","last_fullsized":"2024-04-28T12:50:31.117850+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:55:14.610718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.906082+0000","last_change":"2024-04-28T12:48:33.838037+0000","last_active":"2024-04-28T12:50:30.906082+0000","last_peered":"2024-04-28T12:50:30.906082+0000","last_clean":"2024-04-28T12:50:30.906082+0000","last_became_active":"2024-04-28T12:48:33.837704+0000","last_became_peered":"2024-04-28T12:48:33.837704+0000","last_unstale":"2024-04-28T12:50:30.906082+0000","last_undegraded":"2024-04-28T12:50:30.906082+0000","last_fullsized":"2024-04-28T12:50:30.906082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:46:24.595855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.123235+0000","last_change":"2024-04-28T12:50:31.120476+0000","last_active":"2024-04-28T12:50:31.123235+0000","last_peered":"2024-04-28T12:50:31.123235+0000","last_clean":"2024-04-28T12:50:31.123235+0000","last_became_active":"2024-04-28T12:50:31.119338+0000","last_became_peered":"2024-04-28T12:50:31.119338+0000","last_unstale":"2024-04-28T12:50:31.123235+0000","last_undegraded":"2024-04-28T12:50:31.123235+0000","last_fullsized":"2024-04-28T12:50:31.123235+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:35:07.965270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.120110+0000","last_change":"2024-04-28T12:48:39.615515+0000","last_active":"2024-04-28T12:50:31.120110+0000","last_peered":"2024-04-28T12:50:31.120110+0000","last_clean":"2024-04-28T12:50:31.120110+0000","last_became_active":"2024-04-28T12:48:39.614717+0000","last_became_peered":"2024-04-28T12:48:39.614717+0000","last_unstale":"2024-04-28T12:50:31.120110+0000","last_undegraded":"2024-04-28T12:50:31.120110+0000","last_fullsized":"2024-04-28T12:50:31.120110+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:11:06.633399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887427+0000","last_change":"2024-04-28T12:48:33.832694+0000","last_active":"2024-04-28T12:50:30.887427+0000","last_peered":"2024-04-28T12:50:30.887427+0000","last_clean":"2024-04-28T12:50:30.887427+0000","last_became_active":"2024-04-28T12:48:33.832381+0000","last_became_peered":"2024-04-28T12:48:33.832381+0000","last_unstale":"2024-04-28T12:50:30.887427+0000","last_undegraded":"2024-04-28T12:50:30.887427+0000","last_fullsized":"2024-04-28T12:50:30.887427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:56:57.511053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.865713+0000","last_change":"2024-04-28T12:48:34.560229+0000","last_active":"2024-04-28T12:50:30.865713+0000","last_peered":"2024-04-28T12:50:30.865713+0000","last_clean":"2024-04-28T12:50:30.865713+0000","last_became_active":"2024-04-28T12:48:34.559926+0000","last_became_peered":"2024-04-28T12:48:34.559926+0000","last_unstale":"2024-04-28T12:50:30.865713+0000","last_undegraded":"2024-04-28T12:50:30.865713+0000","last_fullsized":"2024-04-28T12:50:30.865713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:38:22.476449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.860154+0000","last_change":"2024-04-28T12:48:39.604042+0000","last_active":"2024-04-28T12:50:30.860154+0000","last_peered":"2024-04-28T12:50:30.860154+0000","last_clean":"2024-04-28T12:50:30.860154+0000","last_became_active":"2024-04-28T12:48:39.603631+0000","last_became_peered":"2024-04-28T12:48:39.603631+0000","last_unstale":"2024-04-28T12:50:30.860154+0000","last_undegraded":"2024-04-28T12:50:30.860154+0000","last_fullsized":"2024-04-28T12:50:30.860154+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:53:20.073434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":121,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:32.890850+0000","last_change":"2024-04-28T12:48:33.846372+0000","last_active":"2024-04-28T12:50:32.890850+0000","last_peered":"2024-04-28T12:50:32.890850+0000","last_clean":"2024-04-28T12:50:32.890850+0000","last_became_active":"2024-04-28T12:48:33.846138+0000","last_became_peered":"2024-04-28T12:48:33.846138+0000","last_unstale":"2024-04-28T12:50:32.890850+0000","last_undegraded":"2024-04-28T12:50:32.890850+0000","last_fullsized":"2024-04-28T12:50:32.890850+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:46:45.889188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118376+0000","last_change":"2024-04-28T12:48:34.554872+0000","last_active":"2024-04-28T12:50:31.118376+0000","last_peered":"2024-04-28T12:50:31.118376+0000","last_clean":"2024-04-28T12:50:31.118376+0000","last_became_active":"2024-04-28T12:48:34.554555+0000","last_became_peered":"2024-04-28T12:48:34.554555+0000","last_unstale":"2024-04-28T12:50:31.118376+0000","last_undegraded":"2024-04-28T12:50:31.118376+0000","last_fullsized":"2024-04-28T12:50:31.118376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:17:51.280686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.866600+0000","last_change":"2024-04-28T12:48:39.603114+0000","last_active":"2024-04-28T12:50:30.866600+0000","last_peered":"2024-04-28T12:50:30.866600+0000","last_clean":"2024-04-28T12:50:30.866600+0000","last_became_active":"2024-04-28T12:48:39.598404+0000","last_became_peered":"2024-04-28T12:48:39.598404+0000","last_unstale":"2024-04-28T12:50:30.866600+0000","last_undegraded":"2024-04-28T12:50:30.866600+0000","last_fullsized":"2024-04-28T12:50:30.866600+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:23.824317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118002+0000","last_change":"2024-04-28T12:48:33.831626+0000","last_active":"2024-04-28T12:50:31.118002+0000","last_peered":"2024-04-28T12:50:31.118002+0000","last_clean":"2024-04-28T12:50:31.118002+0000","last_became_active":"2024-04-28T12:48:33.831294+0000","last_became_peered":"2024-04-28T12:48:33.831294+0000","last_unstale":"2024-04-28T12:50:31.118002+0000","last_undegraded":"2024-04-28T12:50:31.118002+0000","last_fullsized":"2024-04-28T12:50:31.118002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:58:43.775695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.120242+0000","last_change":"2024-04-28T12:48:34.574987+0000","last_active":"2024-04-28T12:50:31.120242+0000","last_peered":"2024-04-28T12:50:31.120242+0000","last_clean":"2024-04-28T12:50:31.120242+0000","last_became_active":"2024-04-28T12:48:34.574668+0000","last_became_peered":"2024-04-28T12:48:34.574668+0000","last_unstale":"2024-04-28T12:50:31.120242+0000","last_undegraded":"2024-04-28T12:50:31.120242+0000","last_fullsized":"2024-04-28T12:50:31.120242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:24:06.195671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.908966+0000","last_change":"2024-04-28T12:49:30.994058+0000","last_active":"2024-04-28T12:50:30.908966+0000","last_peered":"2024-04-28T12:50:30.908966+0000","last_clean":"2024-04-28T12:50:30.908966+0000","last_became_active":"2024-04-28T12:49:30.993311+0000","last_became_peered":"2024-04-28T12:49:30.993311+0000","last_unstale":"2024-04-28T12:50:30.908966+0000","last_undegraded":"2024-04-28T12:50:30.908966+0000","last_fullsized":"2024-04-28T12:50:30.908966+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:37:08.699751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.888036+0000","last_change":"2024-04-28T12:48:33.840884+0000","last_active":"2024-04-28T12:50:30.888036+0000","last_peered":"2024-04-28T12:50:30.888036+0000","last_clean":"2024-04-28T12:50:30.888036+0000","last_became_active":"2024-04-28T12:48:33.840467+0000","last_became_peered":"2024-04-28T12:48:33.840467+0000","last_unstale":"2024-04-28T12:50:30.888036+0000","last_undegraded":"2024-04-28T12:50:30.888036+0000","last_fullsized":"2024-04-28T12:50:30.888036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:28:16.480520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117926+0000","last_change":"2024-04-28T12:48:34.545000+0000","last_active":"2024-04-28T12:50:31.117926+0000","last_peered":"2024-04-28T12:50:31.117926+0000","last_clean":"2024-04-28T12:50:31.117926+0000","last_became_active":"2024-04-28T12:48:34.544662+0000","last_became_peered":"2024-04-28T12:48:34.544662+0000","last_unstale":"2024-04-28T12:50:31.117926+0000","last_undegraded":"2024-04-28T12:50:31.117926+0000","last_fullsized":"2024-04-28T12:50:31.117926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:19:21.640362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.866287+0000","last_change":"2024-04-28T12:48:39.596755+0000","last_active":"2024-04-28T12:50:30.866287+0000","last_peered":"2024-04-28T12:50:30.866287+0000","last_clean":"2024-04-28T12:50:30.866287+0000","last_became_active":"2024-04-28T12:48:39.596292+0000","last_became_peered":"2024-04-28T12:48:39.596292+0000","last_unstale":"2024-04-28T12:50:30.866287+0000","last_undegraded":"2024-04-28T12:50:30.866287+0000","last_fullsized":"2024-04-28T12:50:30.866287+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:21:27.026940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'32","reported_seq":244,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:33.892088+0000","last_change":"2024-04-28T12:48:33.833840+0000","last_active":"2024-04-28T12:50:33.892088+0000","last_peered":"2024-04-28T12:50:33.892088+0000","last_clean":"2024-04-28T12:50:33.892088+0000","last_became_active":"2024-04-28T12:48:33.833536+0000","last_became_peered":"2024-04-28T12:48:33.833536+0000","last_unstale":"2024-04-28T12:50:33.892088+0000","last_undegraded":"2024-04-28T12:50:33.892088+0000","last_fullsized":"2024-04-28T12:50:33.892088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:37:01.455373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54826,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":180,"num_read_kb":143,"num_write":33,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998494+0000","last_change":"2024-04-28T12:48:34.548046+0000","last_active":"2024-04-28T12:50:30.998494+0000","last_peered":"2024-04-28T12:50:30.998494+0000","last_clean":"2024-04-28T12:50:30.998494+0000","last_became_active":"2024-04-28T12:48:34.547724+0000","last_became_peered":"2024-04-28T12:48:34.547724+0000","last_unstale":"2024-04-28T12:50:30.998494+0000","last_undegraded":"2024-04-28T12:50:30.998494+0000","last_fullsized":"2024-04-28T12:50:30.998494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:56:07.287597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.888331+0000","last_change":"2024-04-28T12:48:39.602976+0000","last_active":"2024-04-28T12:50:30.888331+0000","last_peered":"2024-04-28T12:50:30.888331+0000","last_clean":"2024-04-28T12:50:30.888331+0000","last_became_active":"2024-04-28T12:48:39.602588+0000","last_became_peered":"2024-04-28T12:48:39.602588+0000","last_unstale":"2024-04-28T12:50:30.888331+0000","last_undegraded":"2024-04-28T12:50:30.888331+0000","last_fullsized":"2024-04-28T12:50:30.888331+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:18:19.549434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858203+0000","last_change":"2024-04-28T12:48:33.831276+0000","last_active":"2024-04-28T12:50:30.858203+0000","last_peered":"2024-04-28T12:50:30.858203+0000","last_clean":"2024-04-28T12:50:30.858203+0000","last_became_active":"2024-04-28T12:48:33.830845+0000","last_became_peered":"2024-04-28T12:48:33.830845+0000","last_unstale":"2024-04-28T12:50:30.858203+0000","last_undegraded":"2024-04-28T12:50:30.858203+0000","last_fullsized":"2024-04-28T12:50:30.858203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:21:02.619309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.905332+0000","last_change":"2024-04-28T12:48:34.566767+0000","last_active":"2024-04-28T12:50:30.905332+0000","last_peered":"2024-04-28T12:50:30.905332+0000","last_clean":"2024-04-28T12:50:30.905332+0000","last_became_active":"2024-04-28T12:48:34.566404+0000","last_became_peered":"2024-04-28T12:48:34.566404+0000","last_unstale":"2024-04-28T12:50:30.905332+0000","last_undegraded":"2024-04-28T12:50:30.905332+0000","last_fullsized":"2024-04-28T12:50:30.905332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:49:32.684046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117616+0000","last_change":"2024-04-28T12:48:39.607471+0000","last_active":"2024-04-28T12:50:31.117616+0000","last_peered":"2024-04-28T12:50:31.117616+0000","last_clean":"2024-04-28T12:50:31.117616+0000","last_became_active":"2024-04-28T12:48:39.607040+0000","last_became_peered":"2024-04-28T12:48:39.607040+0000","last_unstale":"2024-04-28T12:50:31.117616+0000","last_undegraded":"2024-04-28T12:50:31.117616+0000","last_fullsized":"2024-04-28T12:50:31.117616+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:06:32.613106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887844+0000","last_change":"2024-04-28T12:48:33.832519+0000","last_active":"2024-04-28T12:50:30.887844+0000","last_peered":"2024-04-28T12:50:30.887844+0000","last_clean":"2024-04-28T12:50:30.887844+0000","last_became_active":"2024-04-28T12:48:33.832146+0000","last_became_peered":"2024-04-28T12:48:33.832146+0000","last_unstale":"2024-04-28T12:50:30.887844+0000","last_undegraded":"2024-04-28T12:50:30.887844+0000","last_fullsized":"2024-04-28T12:50:30.887844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:02:48.755990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118096+0000","last_change":"2024-04-28T12:48:34.557093+0000","last_active":"2024-04-28T12:50:31.118096+0000","last_peered":"2024-04-28T12:50:31.118096+0000","last_clean":"2024-04-28T12:50:31.118096+0000","last_became_active":"2024-04-28T12:48:34.556767+0000","last_became_peered":"2024-04-28T12:48:34.556767+0000","last_unstale":"2024-04-28T12:50:31.118096+0000","last_undegraded":"2024-04-28T12:50:31.118096+0000","last_fullsized":"2024-04-28T12:50:31.118096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:45:48.634998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.868913+0000","last_change":"2024-04-28T12:48:39.612227+0000","last_active":"2024-04-28T12:50:30.868913+0000","last_peered":"2024-04-28T12:50:30.868913+0000","last_clean":"2024-04-28T12:50:30.868913+0000","last_became_active":"2024-04-28T12:48:39.611065+0000","last_became_peered":"2024-04-28T12:48:39.611065+0000","last_unstale":"2024-04-28T12:50:30.868913+0000","last_undegraded":"2024-04-28T12:50:30.868913+0000","last_fullsized":"2024-04-28T12:50:30.868913+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:15:55.833369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'4","reported_seq":127,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:33.915057+0000","last_change":"2024-04-28T12:48:33.835102+0000","last_active":"2024-04-28T12:50:33.915057+0000","last_peered":"2024-04-28T12:50:33.915057+0000","last_clean":"2024-04-28T12:50:33.915057+0000","last_became_active":"2024-04-28T12:48:33.834672+0000","last_became_peered":"2024-04-28T12:48:33.834672+0000","last_unstale":"2024-04-28T12:50:33.915057+0000","last_undegraded":"2024-04-28T12:50:33.915057+0000","last_fullsized":"2024-04-28T12:50:33.915057+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:41:39.388880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117740+0000","last_change":"2024-04-28T12:49:30.707963+0000","last_active":"2024-04-28T12:50:31.117740+0000","last_peered":"2024-04-28T12:50:31.117740+0000","last_clean":"2024-04-28T12:50:31.117740+0000","last_became_active":"2024-04-28T12:49:30.707565+0000","last_became_peered":"2024-04-28T12:49:30.707565+0000","last_unstale":"2024-04-28T12:50:31.117740+0000","last_undegraded":"2024-04-28T12:50:31.117740+0000","last_fullsized":"2024-04-28T12:50:31.117740+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:44:34.238749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.906986+0000","last_change":"2024-04-28T12:48:39.613039+0000","last_active":"2024-04-28T12:50:30.906986+0000","last_peered":"2024-04-28T12:50:30.906986+0000","last_clean":"2024-04-28T12:50:30.906986+0000","last_became_active":"2024-04-28T12:48:39.609383+0000","last_became_peered":"2024-04-28T12:48:39.609383+0000","last_unstale":"2024-04-28T12:50:30.906986+0000","last_undegraded":"2024-04-28T12:50:30.906986+0000","last_fullsized":"2024-04-28T12:50:30.906986+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:55:47.031973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998022+0000","last_change":"2024-04-28T12:48:33.843087+0000","last_active":"2024-04-28T12:50:30.998022+0000","last_peered":"2024-04-28T12:50:30.998022+0000","last_clean":"2024-04-28T12:50:30.998022+0000","last_became_active":"2024-04-28T12:48:33.842775+0000","last_became_peered":"2024-04-28T12:48:33.842775+0000","last_unstale":"2024-04-28T12:50:30.998022+0000","last_undegraded":"2024-04-28T12:50:30.998022+0000","last_fullsized":"2024-04-28T12:50:30.998022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:39:00.797781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887689+0000","last_change":"2024-04-28T12:49:30.993703+0000","last_active":"2024-04-28T12:50:30.887689+0000","last_peered":"2024-04-28T12:50:30.887689+0000","last_clean":"2024-04-28T12:50:30.887689+0000","last_became_active":"2024-04-28T12:49:30.992940+0000","last_became_peered":"2024-04-28T12:49:30.992940+0000","last_unstale":"2024-04-28T12:50:30.887689+0000","last_undegraded":"2024-04-28T12:50:30.887689+0000","last_fullsized":"2024-04-28T12:50:30.887689+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:42:01.384945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887656+0000","last_change":"2024-04-28T12:48:39.608191+0000","last_active":"2024-04-28T12:50:30.887656+0000","last_peered":"2024-04-28T12:50:30.887656+0000","last_clean":"2024-04-28T12:50:30.887656+0000","last_became_active":"2024-04-28T12:48:39.607724+0000","last_became_peered":"2024-04-28T12:48:39.607724+0000","last_unstale":"2024-04-28T12:50:30.887656+0000","last_undegraded":"2024-04-28T12:50:30.887656+0000","last_fullsized":"2024-04-28T12:50:30.887656+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:19:58.040666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998502+0000","last_change":"2024-04-28T12:48:33.839588+0000","last_active":"2024-04-28T12:50:30.998502+0000","last_peered":"2024-04-28T12:50:30.998502+0000","last_clean":"2024-04-28T12:50:30.998502+0000","last_became_active":"2024-04-28T12:48:33.839094+0000","last_became_peered":"2024-04-28T12:48:33.839094+0000","last_unstale":"2024-04-28T12:50:30.998502+0000","last_undegraded":"2024-04-28T12:50:30.998502+0000","last_fullsized":"2024-04-28T12:50:30.998502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:18:35.824206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118047+0000","last_change":"2024-04-28T12:49:30.791511+0000","last_active":"2024-04-28T12:50:31.118047+0000","last_peered":"2024-04-28T12:50:31.118047+0000","last_clean":"2024-04-28T12:50:31.118047+0000","last_became_active":"2024-04-28T12:49:30.791098+0000","last_became_peered":"2024-04-28T12:49:30.791098+0000","last_unstale":"2024-04-28T12:50:31.118047+0000","last_undegraded":"2024-04-28T12:50:31.118047+0000","last_fullsized":"2024-04-28T12:50:31.118047+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:53:02.628572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998138+0000","last_change":"2024-04-28T12:48:39.912611+0000","last_active":"2024-04-28T12:50:30.998138+0000","last_peered":"2024-04-28T12:50:30.998138+0000","last_clean":"2024-04-28T12:50:30.998138+0000","last_became_active":"2024-04-28T12:48:39.912097+0000","last_became_peered":"2024-04-28T12:48:39.912097+0000","last_unstale":"2024-04-28T12:50:30.998138+0000","last_undegraded":"2024-04-28T12:50:30.998138+0000","last_fullsized":"2024-04-28T12:50:30.998138+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:31:38.597920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.867921+0000","last_change":"2024-04-28T12:49:30.708298+0000","last_active":"2024-04-28T12:50:30.867921+0000","last_peered":"2024-04-28T12:50:30.867921+0000","last_clean":"2024-04-28T12:50:30.867921+0000","last_became_active":"2024-04-28T12:49:30.707831+0000","last_became_peered":"2024-04-28T12:49:30.707831+0000","last_unstale":"2024-04-28T12:50:30.867921+0000","last_undegraded":"2024-04-28T12:50:30.867921+0000","last_fullsized":"2024-04-28T12:50:30.867921+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:09:21.049858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.856183+0000","last_change":"2024-04-28T12:48:34.568892+0000","last_active":"2024-04-28T12:50:30.856183+0000","last_peered":"2024-04-28T12:50:30.856183+0000","last_clean":"2024-04-28T12:50:30.856183+0000","last_became_active":"2024-04-28T12:48:34.568581+0000","last_became_peered":"2024-04-28T12:48:34.568581+0000","last_unstale":"2024-04-28T12:50:30.856183+0000","last_undegraded":"2024-04-28T12:50:30.856183+0000","last_fullsized":"2024-04-28T12:50:30.856183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:56:51.661049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.906256+0000","last_change":"2024-04-28T12:48:39.616095+0000","last_active":"2024-04-28T12:50:30.906256+0000","last_peered":"2024-04-28T12:50:30.906256+0000","last_clean":"2024-04-28T12:50:30.906256+0000","last_became_active":"2024-04-28T12:48:39.614891+0000","last_became_peered":"2024-04-28T12:48:39.614891+0000","last_unstale":"2024-04-28T12:50:30.906256+0000","last_undegraded":"2024-04-28T12:50:30.906256+0000","last_fullsized":"2024-04-28T12:50:30.906256+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:01:05.645329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117867+0000","last_change":"2024-04-28T12:48:33.833599+0000","last_active":"2024-04-28T12:50:31.117867+0000","last_peered":"2024-04-28T12:50:31.117867+0000","last_clean":"2024-04-28T12:50:31.117867+0000","last_became_active":"2024-04-28T12:48:33.833062+0000","last_became_peered":"2024-04-28T12:48:33.833062+0000","last_unstale":"2024-04-28T12:50:31.117867+0000","last_undegraded":"2024-04-28T12:50:31.117867+0000","last_fullsized":"2024-04-28T12:50:31.117867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:00:13.094007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.855953+0000","last_change":"2024-04-28T12:48:34.558759+0000","last_active":"2024-04-28T12:50:30.855953+0000","last_peered":"2024-04-28T12:50:30.855953+0000","last_clean":"2024-04-28T12:50:30.855953+0000","last_became_active":"2024-04-28T12:48:34.558413+0000","last_became_peered":"2024-04-28T12:48:34.558413+0000","last_unstale":"2024-04-28T12:50:30.855953+0000","last_undegraded":"2024-04-28T12:50:30.855953+0000","last_fullsized":"2024-04-28T12:50:30.855953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:07:47.106653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.907209+0000","last_change":"2024-04-28T12:48:39.914492+0000","last_active":"2024-04-28T12:50:30.907209+0000","last_peered":"2024-04-28T12:50:30.907209+0000","last_clean":"2024-04-28T12:50:30.907209+0000","last_became_active":"2024-04-28T12:48:39.913113+0000","last_became_peered":"2024-04-28T12:48:39.913113+0000","last_unstale":"2024-04-28T12:50:30.907209+0000","last_undegraded":"2024-04-28T12:50:30.907209+0000","last_fullsized":"2024-04-28T12:50:30.907209+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:49:21.221058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.888040+0000","last_change":"2024-04-28T12:48:34.544121+0000","last_active":"2024-04-28T12:50:30.888040+0000","last_peered":"2024-04-28T12:50:30.888040+0000","last_clean":"2024-04-28T12:50:30.888040+0000","last_became_active":"2024-04-28T12:48:34.543816+0000","last_became_peered":"2024-04-28T12:48:34.543816+0000","last_unstale":"2024-04-28T12:50:30.888040+0000","last_undegraded":"2024-04-28T12:50:30.888040+0000","last_fullsized":"2024-04-28T12:50:30.888040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:53:24.237518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":123,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:33.914392+0000","last_change":"2024-04-28T12:48:33.845996+0000","last_active":"2024-04-28T12:50:33.914392+0000","last_peered":"2024-04-28T12:50:33.914392+0000","last_clean":"2024-04-28T12:50:33.914392+0000","last_became_active":"2024-04-28T12:48:33.845618+0000","last_became_peered":"2024-04-28T12:48:33.845618+0000","last_unstale":"2024-04-28T12:50:33.914392+0000","last_undegraded":"2024-04-28T12:50:33.914392+0000","last_fullsized":"2024-04-28T12:50:33.914392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:53:12.312461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117959+0000","last_change":"2024-04-28T12:48:39.589281+0000","last_active":"2024-04-28T12:50:31.117959+0000","last_peered":"2024-04-28T12:50:31.117959+0000","last_clean":"2024-04-28T12:50:31.117959+0000","last_became_active":"2024-04-28T12:48:39.588964+0000","last_became_peered":"2024-04-28T12:48:39.588964+0000","last_unstale":"2024-04-28T12:50:31.117959+0000","last_undegraded":"2024-04-28T12:50:31.117959+0000","last_fullsized":"2024-04-28T12:50:31.117959+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:32:12.039897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.860817+0000","last_change":"2024-04-28T12:48:34.567062+0000","last_active":"2024-04-28T12:50:30.860817+0000","last_peered":"2024-04-28T12:50:30.860817+0000","last_clean":"2024-04-28T12:50:30.860817+0000","last_became_active":"2024-04-28T12:48:34.565977+0000","last_became_peered":"2024-04-28T12:48:34.565977+0000","last_unstale":"2024-04-28T12:50:30.860817+0000","last_undegraded":"2024-04-28T12:50:30.860817+0000","last_fullsized":"2024-04-28T12:50:30.860817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:46:03.775693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.908912+0000","last_change":"2024-04-28T12:48:33.834069+0000","last_active":"2024-04-28T12:50:30.908912+0000","last_peered":"2024-04-28T12:50:30.908912+0000","last_clean":"2024-04-28T12:50:30.908912+0000","last_became_active":"2024-04-28T12:48:33.833777+0000","last_became_peered":"2024-04-28T12:48:33.833777+0000","last_unstale":"2024-04-28T12:50:30.908912+0000","last_undegraded":"2024-04-28T12:50:30.908912+0000","last_fullsized":"2024-04-28T12:50:30.908912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:32:52.737718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":356,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117523+0000","last_change":"2024-04-28T12:47:13.177589+0000","last_active":"2024-04-28T12:50:31.117523+0000","last_peered":"2024-04-28T12:50:31.117523+0000","last_clean":"2024-04-28T12:50:31.117523+0000","last_became_active":"2024-04-28T12:47:13.177133+0000","last_became_peered":"2024-04-28T12:47:13.177133+0000","last_unstale":"2024-04-28T12:50:31.117523+0000","last_undegraded":"2024-04-28T12:50:31.117523+0000","last_fullsized":"2024-04-28T12:50:31.117523+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:43:15.851674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:43:15.851674+0000","last_clean_scrub_stamp":"2024-04-28T12:43:15.851674+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:23:21.868028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887921+0000","last_change":"2024-04-28T12:49:30.791968+0000","last_active":"2024-04-28T12:50:30.887921+0000","last_peered":"2024-04-28T12:50:30.887921+0000","last_clean":"2024-04-28T12:50:30.887921+0000","last_became_active":"2024-04-28T12:49:30.791380+0000","last_became_peered":"2024-04-28T12:49:30.791380+0000","last_unstale":"2024-04-28T12:50:30.887921+0000","last_undegraded":"2024-04-28T12:50:30.887921+0000","last_fullsized":"2024-04-28T12:50:30.887921+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:56:31.784446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.905166+0000","last_change":"2024-04-28T12:48:34.550976+0000","last_active":"2024-04-28T12:50:30.905166+0000","last_peered":"2024-04-28T12:50:30.905166+0000","last_clean":"2024-04-28T12:50:30.905166+0000","last_became_active":"2024-04-28T12:48:34.549460+0000","last_became_peered":"2024-04-28T12:48:34.549460+0000","last_unstale":"2024-04-28T12:50:30.905166+0000","last_undegraded":"2024-04-28T12:50:30.905166+0000","last_fullsized":"2024-04-28T12:50:30.905166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:55:16.272917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117706+0000","last_change":"2024-04-28T12:48:33.837248+0000","last_active":"2024-04-28T12:50:31.117706+0000","last_peered":"2024-04-28T12:50:31.117706+0000","last_clean":"2024-04-28T12:50:31.117706+0000","last_became_active":"2024-04-28T12:48:33.836910+0000","last_became_peered":"2024-04-28T12:48:33.836910+0000","last_unstale":"2024-04-28T12:50:31.117706+0000","last_undegraded":"2024-04-28T12:50:31.117706+0000","last_fullsized":"2024-04-28T12:50:31.117706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:06:19.572724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117568+0000","last_change":"2024-04-28T12:48:39.605436+0000","last_active":"2024-04-28T12:50:31.117568+0000","last_peered":"2024-04-28T12:50:31.117568+0000","last_clean":"2024-04-28T12:50:31.117568+0000","last_became_active":"2024-04-28T12:48:39.604997+0000","last_became_peered":"2024-04-28T12:48:39.604997+0000","last_unstale":"2024-04-28T12:50:31.117568+0000","last_undegraded":"2024-04-28T12:50:31.117568+0000","last_fullsized":"2024-04-28T12:50:31.117568+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:05:11.571064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998502+0000","last_change":"2024-04-28T12:48:33.848257+0000","last_active":"2024-04-28T12:50:30.998502+0000","last_peered":"2024-04-28T12:50:30.998502+0000","last_clean":"2024-04-28T12:50:30.998502+0000","last_became_active":"2024-04-28T12:48:33.848066+0000","last_became_peered":"2024-04-28T12:48:33.848066+0000","last_unstale":"2024-04-28T12:50:30.998502+0000","last_undegraded":"2024-04-28T12:50:30.998502+0000","last_fullsized":"2024-04-28T12:50:30.998502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:58:34.174153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.907075+0000","last_change":"2024-04-28T12:48:34.559791+0000","last_active":"2024-04-28T12:50:30.907075+0000","last_peered":"2024-04-28T12:50:30.907075+0000","last_clean":"2024-04-28T12:50:30.907075+0000","last_became_active":"2024-04-28T12:48:34.556980+0000","last_became_peered":"2024-04-28T12:48:34.556980+0000","last_unstale":"2024-04-28T12:50:30.907075+0000","last_undegraded":"2024-04-28T12:50:30.907075+0000","last_fullsized":"2024-04-28T12:50:30.907075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:38:56.998159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.860310+0000","last_change":"2024-04-28T12:48:39.917423+0000","last_active":"2024-04-28T12:50:30.860310+0000","last_peered":"2024-04-28T12:50:30.860310+0000","last_clean":"2024-04-28T12:50:30.860310+0000","last_became_active":"2024-04-28T12:48:39.914893+0000","last_became_peered":"2024-04-28T12:48:39.914893+0000","last_unstale":"2024-04-28T12:50:30.860310+0000","last_undegraded":"2024-04-28T12:50:30.860310+0000","last_fullsized":"2024-04-28T12:50:30.860310+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:08:05.499808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'9","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.869240+0000","last_change":"2024-04-28T12:49:30.715567+0000","last_active":"2024-04-28T12:50:30.869240+0000","last_peered":"2024-04-28T12:50:30.869240+0000","last_clean":"2024-04-28T12:50:30.869240+0000","last_became_active":"2024-04-28T12:49:30.711050+0000","last_became_peered":"2024-04-28T12:49:30.711050+0000","last_unstale":"2024-04-28T12:50:30.869240+0000","last_undegraded":"2024-04-28T12:50:30.869240+0000","last_fullsized":"2024-04-28T12:50:30.869240+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T17:01:32.936140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.863738+0000","last_change":"2024-04-28T12:48:34.566120+0000","last_active":"2024-04-28T12:50:30.863738+0000","last_peered":"2024-04-28T12:50:30.863738+0000","last_clean":"2024-04-28T12:50:30.863738+0000","last_became_active":"2024-04-28T12:48:34.564357+0000","last_became_peered":"2024-04-28T12:48:34.564357+0000","last_unstale":"2024-04-28T12:50:30.863738+0000","last_undegraded":"2024-04-28T12:50:30.863738+0000","last_fullsized":"2024-04-28T12:50:30.863738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:28:43.463008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.857836+0000","last_change":"2024-04-28T12:48:39.586362+0000","last_active":"2024-04-28T12:50:30.857836+0000","last_peered":"2024-04-28T12:50:30.857836+0000","last_clean":"2024-04-28T12:50:30.857836+0000","last_became_active":"2024-04-28T12:48:39.585898+0000","last_became_peered":"2024-04-28T12:48:39.585898+0000","last_unstale":"2024-04-28T12:50:30.857836+0000","last_undegraded":"2024-04-28T12:50:30.857836+0000","last_fullsized":"2024-04-28T12:50:30.857836+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:51:19.098797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.909323+0000","last_change":"2024-04-28T12:48:33.833163+0000","last_active":"2024-04-28T12:50:30.909323+0000","last_peered":"2024-04-28T12:50:30.909323+0000","last_clean":"2024-04-28T12:50:30.909323+0000","last_became_active":"2024-04-28T12:48:33.832798+0000","last_became_peered":"2024-04-28T12:48:33.832798+0000","last_unstale":"2024-04-28T12:50:30.909323+0000","last_undegraded":"2024-04-28T12:50:30.909323+0000","last_fullsized":"2024-04-28T12:50:30.909323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:42:07.511346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.120766+0000","last_change":"2024-04-28T12:48:34.576917+0000","last_active":"2024-04-28T12:50:31.120766+0000","last_peered":"2024-04-28T12:50:31.120766+0000","last_clean":"2024-04-28T12:50:31.120766+0000","last_became_active":"2024-04-28T12:48:34.575602+0000","last_became_peered":"2024-04-28T12:48:34.575602+0000","last_unstale":"2024-04-28T12:50:31.120766+0000","last_undegraded":"2024-04-28T12:50:31.120766+0000","last_fullsized":"2024-04-28T12:50:31.120766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:47:21.769208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.909137+0000","last_change":"2024-04-28T12:48:39.913164+0000","last_active":"2024-04-28T12:50:30.909137+0000","last_peered":"2024-04-28T12:50:30.909137+0000","last_clean":"2024-04-28T12:50:30.909137+0000","last_became_active":"2024-04-28T12:48:39.912716+0000","last_became_peered":"2024-04-28T12:48:39.912716+0000","last_unstale":"2024-04-28T12:50:30.909137+0000","last_undegraded":"2024-04-28T12:50:30.909137+0000","last_fullsized":"2024-04-28T12:50:30.909137+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:26:34.544275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117060+0000","last_change":"2024-04-28T12:48:33.831286+0000","last_active":"2024-04-28T12:50:31.117060+0000","last_peered":"2024-04-28T12:50:31.117060+0000","last_clean":"2024-04-28T12:50:31.117060+0000","last_became_active":"2024-04-28T12:48:33.830846+0000","last_became_peered":"2024-04-28T12:48:33.830846+0000","last_unstale":"2024-04-28T12:50:31.117060+0000","last_undegraded":"2024-04-28T12:50:31.117060+0000","last_fullsized":"2024-04-28T12:50:31.117060+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:32:47.985987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998044+0000","last_change":"2024-04-28T12:48:34.543780+0000","last_active":"2024-04-28T12:50:30.998044+0000","last_peered":"2024-04-28T12:50:30.998044+0000","last_clean":"2024-04-28T12:50:30.998044+0000","last_became_active":"2024-04-28T12:48:34.543441+0000","last_became_peered":"2024-04-28T12:48:34.543441+0000","last_unstale":"2024-04-28T12:50:30.998044+0000","last_undegraded":"2024-04-28T12:50:30.998044+0000","last_fullsized":"2024-04-28T12:50:30.998044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:05:43.334523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.869327+0000","last_change":"2024-04-28T12:48:39.607360+0000","last_active":"2024-04-28T12:50:30.869327+0000","last_peered":"2024-04-28T12:50:30.869327+0000","last_clean":"2024-04-28T12:50:30.869327+0000","last_became_active":"2024-04-28T12:48:39.606944+0000","last_became_peered":"2024-04-28T12:48:39.606944+0000","last_unstale":"2024-04-28T12:50:30.869327+0000","last_undegraded":"2024-04-28T12:50:30.869327+0000","last_fullsized":"2024-04-28T12:50:30.869327+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:59:15.020193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.860612+0000","last_change":"2024-04-28T12:48:33.835694+0000","last_active":"2024-04-28T12:50:30.860612+0000","last_peered":"2024-04-28T12:50:30.860612+0000","last_clean":"2024-04-28T12:50:30.860612+0000","last_became_active":"2024-04-28T12:48:33.835210+0000","last_became_peered":"2024-04-28T12:48:33.835210+0000","last_unstale":"2024-04-28T12:50:30.860612+0000","last_undegraded":"2024-04-28T12:50:30.860612+0000","last_fullsized":"2024-04-28T12:50:30.860612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:22:34.657665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117993+0000","last_change":"2024-04-28T12:48:34.556933+0000","last_active":"2024-04-28T12:50:31.117993+0000","last_peered":"2024-04-28T12:50:31.117993+0000","last_clean":"2024-04-28T12:50:31.117993+0000","last_became_active":"2024-04-28T12:48:34.556628+0000","last_became_peered":"2024-04-28T12:48:34.556628+0000","last_unstale":"2024-04-28T12:50:31.117993+0000","last_undegraded":"2024-04-28T12:50:31.117993+0000","last_fullsized":"2024-04-28T12:50:31.117993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-30T00:40:50.464723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.909008+0000","last_change":"2024-04-28T12:48:39.594808+0000","last_active":"2024-04-28T12:50:30.909008+0000","last_peered":"2024-04-28T12:50:30.909008+0000","last_clean":"2024-04-28T12:50:30.909008+0000","last_became_active":"2024-04-28T12:48:39.594333+0000","last_became_peered":"2024-04-28T12:48:39.594333+0000","last_unstale":"2024-04-28T12:50:30.909008+0000","last_undegraded":"2024-04-28T12:50:30.909008+0000","last_fullsized":"2024-04-28T12:50:30.909008+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:34:51.548490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.857973+0000","last_change":"2024-04-28T12:48:33.836878+0000","last_active":"2024-04-28T12:50:30.857973+0000","last_peered":"2024-04-28T12:50:30.857973+0000","last_clean":"2024-04-28T12:50:30.857973+0000","last_became_active":"2024-04-28T12:48:33.836536+0000","last_became_peered":"2024-04-28T12:48:33.836536+0000","last_unstale":"2024-04-28T12:50:30.857973+0000","last_undegraded":"2024-04-28T12:50:30.857973+0000","last_fullsized":"2024-04-28T12:50:30.857973+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:50:41.064854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.861399+0000","last_change":"2024-04-28T12:48:34.575213+0000","last_active":"2024-04-28T12:50:30.861399+0000","last_peered":"2024-04-28T12:50:30.861399+0000","last_clean":"2024-04-28T12:50:30.861399+0000","last_became_active":"2024-04-28T12:48:34.574892+0000","last_became_peered":"2024-04-28T12:48:34.574892+0000","last_unstale":"2024-04-28T12:50:30.861399+0000","last_undegraded":"2024-04-28T12:50:30.861399+0000","last_fullsized":"2024-04-28T12:50:30.861399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:27:54.144215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.909055+0000","last_change":"2024-04-28T12:48:39.912466+0000","last_active":"2024-04-28T12:50:30.909055+0000","last_peered":"2024-04-28T12:50:30.909055+0000","last_clean":"2024-04-28T12:50:30.909055+0000","last_became_active":"2024-04-28T12:48:39.911914+0000","last_became_peered":"2024-04-28T12:48:39.911914+0000","last_unstale":"2024-04-28T12:50:30.909055+0000","last_undegraded":"2024-04-28T12:50:30.909055+0000","last_fullsized":"2024-04-28T12:50:30.909055+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:26:35.341389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.998242+0000","last_change":"2024-04-28T12:48:33.840024+0000","last_active":"2024-04-28T12:50:30.998242+0000","last_peered":"2024-04-28T12:50:30.998242+0000","last_clean":"2024-04-28T12:50:30.998242+0000","last_became_active":"2024-04-28T12:48:33.838608+0000","last_became_peered":"2024-04-28T12:48:33.838608+0000","last_unstale":"2024-04-28T12:50:30.998242+0000","last_undegraded":"2024-04-28T12:50:30.998242+0000","last_fullsized":"2024-04-28T12:50:30.998242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:31:28.060385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118258+0000","last_change":"2024-04-28T12:48:34.554685+0000","last_active":"2024-04-28T12:50:31.118258+0000","last_peered":"2024-04-28T12:50:31.118258+0000","last_clean":"2024-04-28T12:50:31.118258+0000","last_became_active":"2024-04-28T12:48:34.554305+0000","last_became_peered":"2024-04-28T12:48:34.554305+0000","last_unstale":"2024-04-28T12:50:31.118258+0000","last_undegraded":"2024-04-28T12:50:31.118258+0000","last_fullsized":"2024-04-28T12:50:31.118258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T20:19:53.698623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117242+0000","last_change":"2024-04-28T12:48:39.605409+0000","last_active":"2024-04-28T12:50:31.117242+0000","last_peered":"2024-04-28T12:50:31.117242+0000","last_clean":"2024-04-28T12:50:31.117242+0000","last_became_active":"2024-04-28T12:48:39.604991+0000","last_became_peered":"2024-04-28T12:48:39.604991+0000","last_unstale":"2024-04-28T12:50:31.117242+0000","last_undegraded":"2024-04-28T12:50:31.117242+0000","last_fullsized":"2024-04-28T12:50:31.117242+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:37:44.481211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858374+0000","last_change":"2024-04-28T12:48:33.835286+0000","last_active":"2024-04-28T12:50:30.858374+0000","last_peered":"2024-04-28T12:50:30.858374+0000","last_clean":"2024-04-28T12:50:30.858374+0000","last_became_active":"2024-04-28T12:48:33.834729+0000","last_became_peered":"2024-04-28T12:48:33.834729+0000","last_unstale":"2024-04-28T12:50:30.858374+0000","last_undegraded":"2024-04-28T12:50:30.858374+0000","last_fullsized":"2024-04-28T12:50:30.858374+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:52:54.889021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118286+0000","last_change":"2024-04-28T12:48:34.547865+0000","last_active":"2024-04-28T12:50:31.118286+0000","last_peered":"2024-04-28T12:50:31.118286+0000","last_clean":"2024-04-28T12:50:31.118286+0000","last_became_active":"2024-04-28T12:48:34.547226+0000","last_became_peered":"2024-04-28T12:48:34.547226+0000","last_unstale":"2024-04-28T12:50:31.118286+0000","last_undegraded":"2024-04-28T12:50:31.118286+0000","last_fullsized":"2024-04-28T12:50:31.118286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:14:06.376908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.120203+0000","last_change":"2024-04-28T12:49:30.994601+0000","last_active":"2024-04-28T12:50:31.120203+0000","last_peered":"2024-04-28T12:50:31.120203+0000","last_clean":"2024-04-28T12:50:31.120203+0000","last_became_active":"2024-04-28T12:49:30.993830+0000","last_became_peered":"2024-04-28T12:49:30.993830+0000","last_unstale":"2024-04-28T12:50:31.120203+0000","last_undegraded":"2024-04-28T12:50:31.120203+0000","last_fullsized":"2024-04-28T12:50:31.120203+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T12:55:59.131768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.888203+0000","last_change":"2024-04-28T12:48:33.835109+0000","last_active":"2024-04-28T12:50:30.888203+0000","last_peered":"2024-04-28T12:50:30.888203+0000","last_clean":"2024-04-28T12:50:30.888203+0000","last_became_active":"2024-04-28T12:48:33.834757+0000","last_became_peered":"2024-04-28T12:48:33.834757+0000","last_unstale":"2024-04-28T12:50:30.888203+0000","last_undegraded":"2024-04-28T12:50:30.888203+0000","last_fullsized":"2024-04-28T12:50:30.888203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:46:45.884261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118322+0000","last_change":"2024-04-28T12:48:34.565135+0000","last_active":"2024-04-28T12:50:31.118322+0000","last_peered":"2024-04-28T12:50:31.118322+0000","last_clean":"2024-04-28T12:50:31.118322+0000","last_became_active":"2024-04-28T12:48:34.564631+0000","last_became_peered":"2024-04-28T12:48:34.564631+0000","last_unstale":"2024-04-28T12:50:31.118322+0000","last_undegraded":"2024-04-28T12:50:31.118322+0000","last_fullsized":"2024-04-28T12:50:31.118322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:36:46.747085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.855449+0000","last_change":"2024-04-28T12:49:30.993743+0000","last_active":"2024-04-28T12:50:30.855449+0000","last_peered":"2024-04-28T12:50:30.855449+0000","last_clean":"2024-04-28T12:50:30.855449+0000","last_became_active":"2024-04-28T12:49:30.993136+0000","last_became_peered":"2024-04-28T12:49:30.993136+0000","last_unstale":"2024-04-28T12:50:30.855449+0000","last_undegraded":"2024-04-28T12:50:30.855449+0000","last_fullsized":"2024-04-28T12:50:30.855449+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:38.543753+0000","last_clean_scrub_stamp":"2024-04-28T12:48:38.543753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:53:32.903663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,7,1],"acting":[10,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.907114+0000","last_change":"2024-04-28T12:48:33.835565+0000","last_active":"2024-04-28T12:50:30.907114+0000","last_peered":"2024-04-28T12:50:30.907114+0000","last_clean":"2024-04-28T12:50:30.907114+0000","last_became_active":"2024-04-28T12:48:33.835147+0000","last_became_peered":"2024-04-28T12:48:33.835147+0000","last_unstale":"2024-04-28T12:50:30.907114+0000","last_undegraded":"2024-04-28T12:50:30.907114+0000","last_fullsized":"2024-04-28T12:50:30.907114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:32.801493+0000","last_clean_scrub_stamp":"2024-04-28T12:48:32.801493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:48:23.520295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118359+0000","last_change":"2024-04-28T12:48:34.547561+0000","last_active":"2024-04-28T12:50:31.118359+0000","last_peered":"2024-04-28T12:50:31.118359+0000","last_clean":"2024-04-28T12:50:31.118359+0000","last_became_active":"2024-04-28T12:48:34.547236+0000","last_became_peered":"2024-04-28T12:48:34.547236+0000","last_unstale":"2024-04-28T12:50:31.118359+0000","last_undegraded":"2024-04-28T12:50:31.118359+0000","last_fullsized":"2024-04-28T12:50:31.118359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:00:37.150080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.118019+0000","last_change":"2024-04-28T12:48:34.546027+0000","last_active":"2024-04-28T12:50:31.118019+0000","last_peered":"2024-04-28T12:50:31.118019+0000","last_clean":"2024-04-28T12:50:31.118019+0000","last_became_active":"2024-04-28T12:48:34.545618+0000","last_became_peered":"2024-04-28T12:48:34.545618+0000","last_unstale":"2024-04-28T12:50:31.118019+0000","last_undegraded":"2024-04-28T12:50:31.118019+0000","last_fullsized":"2024-04-28T12:50:31.118019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:40:00.928586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858151+0000","last_change":"2024-04-28T12:48:34.544883+0000","last_active":"2024-04-28T12:50:30.858151+0000","last_peered":"2024-04-28T12:50:30.858151+0000","last_clean":"2024-04-28T12:50:30.858151+0000","last_became_active":"2024-04-28T12:48:34.544454+0000","last_became_peered":"2024-04-28T12:48:34.544454+0000","last_unstale":"2024-04-28T12:50:30.858151+0000","last_undegraded":"2024-04-28T12:50:30.858151+0000","last_fullsized":"2024-04-28T12:50:30.858151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:58:16.064432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.857917+0000","last_change":"2024-04-28T12:48:34.565110+0000","last_active":"2024-04-28T12:50:30.857917+0000","last_peered":"2024-04-28T12:50:30.857917+0000","last_clean":"2024-04-28T12:50:30.857917+0000","last_became_active":"2024-04-28T12:48:34.564723+0000","last_became_peered":"2024-04-28T12:48:34.564723+0000","last_unstale":"2024-04-28T12:50:30.857917+0000","last_undegraded":"2024-04-28T12:50:30.857917+0000","last_fullsized":"2024-04-28T12:50:30.857917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T18:58:24.220515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.121000+0000","last_change":"2024-04-28T12:48:34.572428+0000","last_active":"2024-04-28T12:50:31.121000+0000","last_peered":"2024-04-28T12:50:31.121000+0000","last_clean":"2024-04-28T12:50:31.121000+0000","last_became_active":"2024-04-28T12:48:34.572034+0000","last_became_peered":"2024-04-28T12:48:34.572034+0000","last_unstale":"2024-04-28T12:50:31.121000+0000","last_undegraded":"2024-04-28T12:50:31.121000+0000","last_fullsized":"2024-04-28T12:50:31.121000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:38:23.598568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.906819+0000","last_change":"2024-04-28T12:48:34.562425+0000","last_active":"2024-04-28T12:50:30.906819+0000","last_peered":"2024-04-28T12:50:30.906819+0000","last_clean":"2024-04-28T12:50:30.906819+0000","last_became_active":"2024-04-28T12:48:34.562087+0000","last_became_peered":"2024-04-28T12:48:34.562087+0000","last_unstale":"2024-04-28T12:50:30.906819+0000","last_undegraded":"2024-04-28T12:50:30.906819+0000","last_fullsized":"2024-04-28T12:50:30.906819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T23:32:39.701796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117894+0000","last_change":"2024-04-28T12:48:34.547755+0000","last_active":"2024-04-28T12:50:31.117894+0000","last_peered":"2024-04-28T12:50:31.117894+0000","last_clean":"2024-04-28T12:50:31.117894+0000","last_became_active":"2024-04-28T12:48:34.547427+0000","last_became_peered":"2024-04-28T12:48:34.547427+0000","last_unstale":"2024-04-28T12:50:31.117894+0000","last_undegraded":"2024-04-28T12:50:31.117894+0000","last_fullsized":"2024-04-28T12:50:31.117894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T16:58:15.694252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.858531+0000","last_change":"2024-04-28T12:48:34.543146+0000","last_active":"2024-04-28T12:50:30.858531+0000","last_peered":"2024-04-28T12:50:30.858531+0000","last_clean":"2024-04-28T12:50:30.858531+0000","last_became_active":"2024-04-28T12:48:34.542809+0000","last_became_peered":"2024-04-28T12:48:34.542809+0000","last_unstale":"2024-04-28T12:50:30.858531+0000","last_undegraded":"2024-04-28T12:50:30.858531+0000","last_fullsized":"2024-04-28T12:50:30.858531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T13:09:42.515358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.863968+0000","last_change":"2024-04-28T12:48:34.571166+0000","last_active":"2024-04-28T12:50:30.863968+0000","last_peered":"2024-04-28T12:50:30.863968+0000","last_clean":"2024-04-28T12:50:30.863968+0000","last_became_active":"2024-04-28T12:48:34.567803+0000","last_became_peered":"2024-04-28T12:48:34.567803+0000","last_unstale":"2024-04-28T12:50:30.863968+0000","last_undegraded":"2024-04-28T12:50:30.863968+0000","last_fullsized":"2024-04-28T12:50:30.863968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T14:11:09.945751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.887491+0000","last_change":"2024-04-28T12:48:34.564939+0000","last_active":"2024-04-28T12:50:30.887491+0000","last_peered":"2024-04-28T12:50:30.887491+0000","last_clean":"2024-04-28T12:50:30.887491+0000","last_became_active":"2024-04-28T12:48:34.564486+0000","last_became_peered":"2024-04-28T12:48:34.564486+0000","last_unstale":"2024-04-28T12:50:30.887491+0000","last_undegraded":"2024-04-28T12:50:30.887491+0000","last_fullsized":"2024-04-28T12:50:30.887491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:29:44.618469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.860774+0000","last_change":"2024-04-28T12:48:34.567359+0000","last_active":"2024-04-28T12:50:30.860774+0000","last_peered":"2024-04-28T12:50:30.860774+0000","last_clean":"2024-04-28T12:50:30.860774+0000","last_became_active":"2024-04-28T12:48:34.565393+0000","last_became_peered":"2024-04-28T12:48:34.565393+0000","last_unstale":"2024-04-28T12:50:30.860774+0000","last_undegraded":"2024-04-28T12:50:30.860774+0000","last_fullsized":"2024-04-28T12:50:30.860774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T22:44:04.312257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117592+0000","last_change":"2024-04-28T12:48:34.565019+0000","last_active":"2024-04-28T12:50:31.117592+0000","last_peered":"2024-04-28T12:50:31.117592+0000","last_clean":"2024-04-28T12:50:31.117592+0000","last_became_active":"2024-04-28T12:48:34.564635+0000","last_became_peered":"2024-04-28T12:48:34.564635+0000","last_unstale":"2024-04-28T12:50:31.117592+0000","last_undegraded":"2024-04-28T12:50:31.117592+0000","last_fullsized":"2024-04-28T12:50:31.117592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T19:14:20.645281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:30.906591+0000","last_change":"2024-04-28T12:48:34.574043+0000","last_active":"2024-04-28T12:50:30.906591+0000","last_peered":"2024-04-28T12:50:30.906591+0000","last_clean":"2024-04-28T12:50:30.906591+0000","last_became_active":"2024-04-28T12:48:34.573686+0000","last_became_peered":"2024-04-28T12:48:34.573686+0000","last_unstale":"2024-04-28T12:50:30.906591+0000","last_undegraded":"2024-04-28T12:50:30.906591+0000","last_fullsized":"2024-04-28T12:50:30.906591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T21:29:57.371318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-28T12:50:31.117689+0000","last_change":"2024-04-28T12:49:30.710555+0000","last_active":"2024-04-28T12:50:31.117689+0000","last_peered":"2024-04-28T12:50:31.117689+0000","last_clean":"2024-04-28T12:50:31.117689+0000","last_became_active":"2024-04-28T12:49:30.710205+0000","last_became_peered":"2024-04-28T12:49:30.710205+0000","last_unstale":"2024-04-28T12:50:31.117689+0000","last_undegraded":"2024-04-28T12:50:31.117689+0000","last_fullsized":"2024-04-28T12:50:31.117689+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-28T12:48:33.812945+0000","last_clean_scrub_stamp":"2024-04-28T12:48:33.812945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-29T15:40:53.390140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":281028,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59218,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":546,"num_read_kb":336,"num_write":164,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":9,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":177654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":527681,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1171456,"data_stored":1170166,"data_compressed":62465,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808883,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":532013,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1048576,"data_stored":527837,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1097728,"data_stored":1124137,"data_compressed":62465,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":523093,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1089536,"data_stored":577829,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":524481,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1114112,"data_stored":579570,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182486,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":523287,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":528134,"data_compressed":26720,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1105920,"data_stored":1119334,"data_compressed":62465,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34101,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34101,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34101,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35745,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":56567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":56795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-28T12:50:41.521 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-28T12:50:41.522 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-28T12:50:41.522 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-28T12:50:41.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-28T12:50:41.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:41 smithi161 ceph-mon[62387]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:50:41.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:41 smithi161 ceph-mon[62387]: pgmap v390: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 94 B/s wr, 5 op/s 2024-04-28T12:50:41.996 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:42.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:41 smithi084 ceph-mon[28081]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:50:42.141 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:41 smithi084 ceph-mon[28081]: pgmap v390: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 94 B/s wr, 5 op/s 2024-04-28T12:50:42.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:41 smithi179 ceph-mon[63029]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-28T12:50:42.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:41 smithi179 ceph-mon[63029]: pgmap v390: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 94 B/s wr, 5 op/s 2024-04-28T12:50:43.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:43 smithi161 ceph-mon[62387]: pgmap v391: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T12:50:44.009 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:43 smithi084 ceph-mon[28081]: pgmap v391: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T12:50:44.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:43 smithi179 ceph-mon[63029]: pgmap v391: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-28T12:50:44.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.364+0000 7ff0cde00700 1 -- 172.21.15.84:0/1674910660 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c806c2e0 msgr2=0x7ff0c810f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:44.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.364+0000 7ff0cde00700 1 --2- 172.21.15.84:0/1674910660 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c806c2e0 0x7ff0c810f980 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4009990 tx=0x7ff0b4009ca0 comp rx=0 tx=0).stop 2024-04-28T12:50:44.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.364+0000 7ff0cde00700 1 -- 172.21.15.84:0/1674910660 shutdown_connections 2024-04-28T12:50:44.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.364+0000 7ff0cde00700 1 --2- 172.21.15.84:0/1674910660 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0c810fec0 0x7ff0c81122b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:44.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.364+0000 7ff0cde00700 1 --2- 172.21.15.84:0/1674910660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0c806b9c0 0x7ff0c806bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:44.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.364+0000 7ff0cde00700 1 --2- 172.21.15.84:0/1674910660 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c806c2e0 0x7ff0c810f980 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:44.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.364+0000 7ff0cde00700 1 -- 172.21.15.84:0/1674910660 >> 172.21.15.84:0/1674910660 conn(0x7ff0c8079290 msgr2=0x7ff0c80796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:44.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.364+0000 7ff0cde00700 1 -- 172.21.15.84:0/1674910660 shutdown_connections 2024-04-28T12:50:44.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.364+0000 7ff0cde00700 1 -- 172.21.15.84:0/1674910660 wait complete. 2024-04-28T12:50:44.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.365+0000 7ff0cde00700 1 Processor -- start 2024-04-28T12:50:44.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.365+0000 7ff0cde00700 1 -- start start 2024-04-28T12:50:44.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.366+0000 7ff0cde00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0c806b9c0 0x7ff0c819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:44.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.366+0000 7ff0cde00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c806c2e0 0x7ff0c819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:44.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.366+0000 7ff0cde00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0c810fec0 0x7ff0c81a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:44.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.366+0000 7ff0cde00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c8115840 con 0x7ff0c806c2e0 2024-04-28T12:50:44.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.366+0000 7ff0cde00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c81156c0 con 0x7ff0c806b9c0 2024-04-28T12:50:44.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.366+0000 7ff0cde00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c81159c0 con 0x7ff0c810fec0 2024-04-28T12:50:44.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.366+0000 7ff0c6000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c806c2e0 0x7ff0c819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:44.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.366+0000 7ff0c6000700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c806c2e0 0x7ff0c819c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55432/0 (socket says 172.21.15.84:55432) 2024-04-28T12:50:44.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.366+0000 7ff0c6000700 1 -- 172.21.15.84:0/2976380362 learned_addr learned my addr 172.21.15.84:0/2976380362 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:44.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.366+0000 7ff0c7400700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0c810fec0 0x7ff0c81a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:44.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.366+0000 7ff0c6a00700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0c806b9c0 0x7ff0c819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:44.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.367+0000 7ff0c6000700 1 -- 172.21.15.84:0/2976380362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0c810fec0 msgr2=0x7ff0c81a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:44.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.367+0000 7ff0c6000700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0c810fec0 0x7ff0c81a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:44.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.367+0000 7ff0c6000700 1 -- 172.21.15.84:0/2976380362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0c806b9c0 msgr2=0x7ff0c819bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:44.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.367+0000 7ff0c6000700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0c806b9c0 0x7ff0c819bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:44.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.367+0000 7ff0c6000700 1 -- 172.21.15.84:0/2976380362 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c81a0d50 con 0x7ff0c806c2e0 2024-04-28T12:50:44.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.367+0000 7ff0c6a00700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0c806b9c0 0x7ff0c819bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:44.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.367+0000 7ff0c6000700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c806c2e0 0x7ff0c819c2d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4000c00 tx=0x7ff0b40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:44.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.367+0000 7ff0cca00700 1 -- 172.21.15.84:0/2976380362 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4027070 con 0x7ff0c806c2e0 2024-04-28T12:50:44.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.367+0000 7ff0cde00700 1 -- 172.21.15.84:0/2976380362 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c81a0fe0 con 0x7ff0c806c2e0 2024-04-28T12:50:44.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.367+0000 7ff0cca00700 1 -- 172.21.15.84:0/2976380362 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0b4022470 con 0x7ff0c806c2e0 2024-04-28T12:50:44.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.367+0000 7ff0cde00700 1 -- 172.21.15.84:0/2976380362 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c81a88c0 con 0x7ff0c806c2e0 2024-04-28T12:50:44.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.367+0000 7ff0cca00700 1 -- 172.21.15.84:0/2976380362 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b40175f0 con 0x7ff0c806c2e0 2024-04-28T12:50:44.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.369+0000 7ff0cde00700 1 -- 172.21.15.84:0/2976380362 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c806a110 con 0x7ff0c806c2e0 2024-04-28T12:50:44.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.370+0000 7ff0cca00700 1 -- 172.21.15.84:0/2976380362 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff0b402a070 con 0x7ff0c806c2e0 2024-04-28T12:50:44.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.372+0000 7ff0cca00700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff0ac0a3cd0 0x7ff0ac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:44.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.372+0000 7ff0cca00700 1 -- 172.21.15.84:0/2976380362 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff0b40d1670 con 0x7ff0c806c2e0 2024-04-28T12:50:44.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.372+0000 7ff0c6a00700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff0ac0a3cd0 0x7ff0ac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:44.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.373+0000 7ff0c6a00700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff0ac0a3cd0 0x7ff0ac0a6190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8011490 tx=0x7ff0b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:44.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.377+0000 7ff0cca00700 1 -- 172.21.15.84:0/2976380362 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff0b4013070 con 0x7ff0c806c2e0 2024-04-28T12:50:44.645 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.648+0000 7ff0cde00700 1 -- 172.21.15.84:0/2976380362 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff0c802e490 con 0x7ff0c806c2e0 2024-04-28T12:50:44.646 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.649+0000 7ff0cca00700 1 -- 172.21.15.84:0/2976380362 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7ff0b409dba0 con 0x7ff0c806c2e0 2024-04-28T12:50:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:50:44.646 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-28T12:50:44.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.651+0000 7ff0cde00700 1 -- 172.21.15.84:0/2976380362 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff0ac0a3cd0 msgr2=0x7ff0ac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:44.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.651+0000 7ff0cde00700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff0ac0a3cd0 0x7ff0ac0a6190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8011490 tx=0x7ff0b8009260 comp rx=0 tx=0).stop 2024-04-28T12:50:44.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.651+0000 7ff0cde00700 1 -- 172.21.15.84:0/2976380362 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c806c2e0 msgr2=0x7ff0c819c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:44.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.651+0000 7ff0cde00700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c806c2e0 0x7ff0c819c2d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4000c00 tx=0x7ff0b40057c0 comp rx=0 tx=0).stop 2024-04-28T12:50:44.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.651+0000 7ff0cde00700 1 -- 172.21.15.84:0/2976380362 shutdown_connections 2024-04-28T12:50:44.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.651+0000 7ff0cde00700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff0ac0a3cd0 0x7ff0ac0a6190 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:44.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.651+0000 7ff0cde00700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0c806b9c0 0x7ff0c819bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:44.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.651+0000 7ff0cde00700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff0c810fec0 0x7ff0c81a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:44.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.651+0000 7ff0cde00700 1 --2- 172.21.15.84:0/2976380362 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0c806c2e0 0x7ff0c819c2d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:44.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.651+0000 7ff0cde00700 1 -- 172.21.15.84:0/2976380362 >> 172.21.15.84:0/2976380362 conn(0x7ff0c8079290 msgr2=0x7ff0c81113b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:44.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.652+0000 7ff0cde00700 1 -- 172.21.15.84:0/2976380362 shutdown_connections 2024-04-28T12:50:44.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:44.652+0000 7ff0cde00700 1 -- 172.21.15.84:0/2976380362 wait complete. 2024-04-28T12:50:44.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:44 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2976380362' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-28T12:50:44.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:44 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2976380362' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-28T12:50:45.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:44 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2976380362' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-28T12:50:45.563 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-28T12:50:45.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:50:45.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.782+0000 7f25a3800640 1 -- 172.21.15.84:0/3515626938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f259c10ab40 msgr2=0x7f259c1113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:45.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.782+0000 7f25a3800640 1 --2- 172.21.15.84:0/3515626938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f259c10ab40 0x7f259c1113d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2594009ca0 tx=0x7f259402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:45.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.782+0000 7f25a3800640 1 -- 172.21.15.84:0/3515626938 shutdown_connections 2024-04-28T12:50:45.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.782+0000 7f25a3800640 1 --2- 172.21.15.84:0/3515626938 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f259c10ab40 0x7f259c1113d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:45.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.782+0000 7f25a3800640 1 --2- 172.21.15.84:0/3515626938 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f259c10a180 0x7f259c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:45.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.782+0000 7f25a3800640 1 --2- 172.21.15.84:0/3515626938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f259c108f90 0x7f259c1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:45.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.782+0000 7f25a3800640 1 -- 172.21.15.84:0/3515626938 >> 172.21.15.84:0/3515626938 conn(0x7f259c077f40 msgr2=0x7f259c07a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:45.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.782+0000 7f25a3800640 1 -- 172.21.15.84:0/3515626938 shutdown_connections 2024-04-28T12:50:45.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.782+0000 7f25a3800640 1 -- 172.21.15.84:0/3515626938 wait complete. 2024-04-28T12:50:45.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.783+0000 7f25a3800640 1 Processor -- start 2024-04-28T12:50:45.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.783+0000 7f25a3800640 1 -- start start 2024-04-28T12:50:45.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.783+0000 7f25a3800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f259c108f90 0x7f259c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:45.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.783+0000 7f25a3800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f259c10a180 0x7f259c19f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:45.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.783+0000 7f25a3800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f259c10ab40 0x7f259c1a66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:45.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.783+0000 7f25a3800640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f259c114ad0 con 0x7f259c10a180 2024-04-28T12:50:45.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.783+0000 7f25a3800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f259c114950 con 0x7f259c10ab40 2024-04-28T12:50:45.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.783+0000 7f25a3800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f259c114c50 con 0x7f259c108f90 2024-04-28T12:50:45.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.783+0000 7f259be00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f259c10a180 0x7f259c19f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:45.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.784+0000 7f259be00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f259c10a180 0x7f259c19f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55456/0 (socket says 172.21.15.84:55456) 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.784+0000 7f259be00640 1 -- 172.21.15.84:0/4114129520 learned_addr learned my addr 172.21.15.84:0/4114129520 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.784+0000 7f25a0e00640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f259c108f90 0x7f259c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.784+0000 7f25a1800640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f259c10ab40 0x7f259c1a66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.784+0000 7f259be00640 1 -- 172.21.15.84:0/4114129520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f259c108f90 msgr2=0x7f259c19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.784+0000 7f259be00640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f259c108f90 0x7f259c19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.784+0000 7f259be00640 1 -- 172.21.15.84:0/4114129520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f259c10ab40 msgr2=0x7f259c1a66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.784+0000 7f259be00640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f259c10ab40 0x7f259c1a66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.784+0000 7f259be00640 1 -- 172.21.15.84:0/4114129520 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f259c1a6d40 con 0x7f259c10a180 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.784+0000 7f259be00640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f259c10a180 0x7f259c19f660 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f258800e9f0 tx=0x7f258800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.785+0000 7f2599600640 1 -- 172.21.15.84:0/4114129520 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258800cd80 con 0x7f259c10a180 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.785+0000 7f2599600640 1 -- 172.21.15.84:0/4114129520 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25880108e0 con 0x7f259c10a180 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.785+0000 7f25a3800640 1 -- 172.21.15.84:0/4114129520 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f259c1a6fd0 con 0x7f259c10a180 2024-04-28T12:50:45.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.785+0000 7f2599600640 1 -- 172.21.15.84:0/4114129520 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2588018980 con 0x7f259c10a180 2024-04-28T12:50:45.784 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.785+0000 7f25a3800640 1 -- 172.21.15.84:0/4114129520 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f259c1a74f0 con 0x7f259c10a180 2024-04-28T12:50:45.784 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.786+0000 7f25a3800640 1 -- 172.21.15.84:0/4114129520 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f259c1102b0 con 0x7f259c10a180 2024-04-28T12:50:45.785 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.788+0000 7f2599600640 1 -- 172.21.15.84:0/4114129520 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2588010430 con 0x7f259c10a180 2024-04-28T12:50:45.786 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.790+0000 7f2599600640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f25800a3c40 0x7f25800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:45.787 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.790+0000 7f2599600640 1 -- 172.21.15.84:0/4114129520 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f25880d0c20 con 0x7f259c10a180 2024-04-28T12:50:45.787 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.790+0000 7f25a0e00640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f25800a3c40 0x7f25800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:45.787 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.791+0000 7f25a0e00640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f25800a3c40 0x7f25800a6100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f259c109fe0 tx=0x7f258c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:45.791 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.794+0000 7f2599600640 1 -- 172.21.15.84:0/4114129520 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f258809ca70 con 0x7f259c10a180 2024-04-28T12:50:45.995 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:45.998+0000 7f25a3800640 1 -- 172.21.15.84:0/4114129520 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f259c02dd20 con 0x7f259c10a180 2024-04-28T12:50:45.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.001+0000 7f2599600640 1 -- 172.21.15.84:0/4114129520 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24634 (secure 0 0 0) 0x7f25880a0d70 con 0x7f259c10a180 2024-04-28T12:50:45.999 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:50:45.999 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":94,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:50:30.848698+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.8","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:50:46.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.004+0000 7f25a3800640 1 -- 172.21.15.84:0/4114129520 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f25800a3c40 msgr2=0x7f25800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:46.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.004+0000 7f25a3800640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f25800a3c40 0x7f25800a6100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f259c109fe0 tx=0x7f258c005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:46.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.004+0000 7f25a3800640 1 -- 172.21.15.84:0/4114129520 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f259c10a180 msgr2=0x7f259c19f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:46.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.004+0000 7f25a3800640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f259c10a180 0x7f259c19f660 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f258800e9f0 tx=0x7f258800eeb0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.004+0000 7f25a3800640 1 -- 172.21.15.84:0/4114129520 shutdown_connections 2024-04-28T12:50:46.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.004+0000 7f25a3800640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f25800a3c40 0x7f25800a6100 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.004+0000 7f25a3800640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f259c10ab40 0x7f259c1a66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.004+0000 7f25a3800640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f259c10a180 0x7f259c19f660 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.004+0000 7f25a3800640 1 --2- 172.21.15.84:0/4114129520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f259c108f90 0x7f259c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.004+0000 7f25a3800640 1 -- 172.21.15.84:0/4114129520 >> 172.21.15.84:0/4114129520 conn(0x7f259c077f40 msgr2=0x7f259c10d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:46.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.004+0000 7f25a3800640 1 -- 172.21.15.84:0/4114129520 shutdown_connections 2024-04-28T12:50:46.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.004+0000 7f25a3800640 1 -- 172.21.15.84:0/4114129520 wait complete. 2024-04-28T12:50:46.018 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:45 smithi084 ceph-mon[28081]: pgmap v392: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:46.021 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:43:15.608080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T12:48:32.037809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T12:48:33.240166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T12:48:37.567238+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T12:50:46.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:50:46.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:45 smithi179 ceph-mon[63029]: pgmap v392: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:46.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.224+0000 7f6a69800640 1 -- 172.21.15.84:0/3520415556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a64103ab0 msgr2=0x7f6a64103f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:46.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.224+0000 7f6a69800640 1 --2- 172.21.15.84:0/3520415556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a64103ab0 0x7f6a64103f30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009900 tx=0x7f6a5002a3f0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.224+0000 7f6a69800640 1 -- 172.21.15.84:0/3520415556 shutdown_connections 2024-04-28T12:50:46.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.224+0000 7f6a69800640 1 --2- 172.21.15.84:0/3520415556 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a64104470 0x7f6a6410ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.224+0000 7f6a69800640 1 --2- 172.21.15.84:0/3520415556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a64103ab0 0x7f6a64103f30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.224+0000 7f6a69800640 1 --2- 172.21.15.84:0/3520415556 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a641028c0 0x7f6a64102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.224+0000 7f6a69800640 1 -- 172.21.15.84:0/3520415556 >> 172.21.15.84:0/3520415556 conn(0x7f6a640fde30 msgr2=0x7f6a64100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:46.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.224+0000 7f6a69800640 1 -- 172.21.15.84:0/3520415556 shutdown_connections 2024-04-28T12:50:46.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.224+0000 7f6a69800640 1 -- 172.21.15.84:0/3520415556 wait complete. 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.225+0000 7f6a69800640 1 Processor -- start 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.226+0000 7f6a69800640 1 -- start start 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.226+0000 7f6a69800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a641028c0 0x7f6a6419ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.226+0000 7f6a69800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a64103ab0 0x7f6a6419b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.226+0000 7f6a69800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a64104470 0x7f6a641a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.226+0000 7f6a69800640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6410e3d0 con 0x7f6a64103ab0 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.226+0000 7f6a69800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6410e250 con 0x7f6a64104470 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.226+0000 7f6a69800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6410e550 con 0x7f6a641028c0 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.226+0000 7f6a63e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a641028c0 0x7f6a6419ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.226+0000 7f6a63e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a641028c0 0x7f6a6419ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:45316/0 (socket says 172.21.15.84:45316) 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.226+0000 7f6a63400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a64103ab0 0x7f6a6419b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.226+0000 7f6a63e00640 1 -- 172.21.15.84:0/777607884 learned_addr learned my addr 172.21.15.84:0/777607884 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:46.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.226+0000 7f6a68e00640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a64104470 0x7f6a641a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:46.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.227+0000 7f6a63e00640 1 -- 172.21.15.84:0/777607884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a64104470 msgr2=0x7f6a641a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:46.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.227+0000 7f6a63e00640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a64104470 0x7f6a641a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.227+0000 7f6a63e00640 1 -- 172.21.15.84:0/777607884 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a64103ab0 msgr2=0x7f6a6419b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:46.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.227+0000 7f6a63e00640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a64103ab0 0x7f6a6419b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.227+0000 7f6a63e00640 1 -- 172.21.15.84:0/777607884 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a641a28c0 con 0x7f6a641028c0 2024-04-28T12:50:46.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.227+0000 7f6a63e00640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a641028c0 0x7f6a6419ac80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6a5400e9c0 tx=0x7f6a5400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:46.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.227+0000 7f6a60c00640 1 -- 172.21.15.84:0/777607884 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5400cd80 con 0x7f6a641028c0 2024-04-28T12:50:46.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.227+0000 7f6a60c00640 1 -- 172.21.15.84:0/777607884 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a540035f0 con 0x7f6a641028c0 2024-04-28T12:50:46.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.227+0000 7f6a60c00640 1 -- 172.21.15.84:0/777607884 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54010690 con 0x7f6a641028c0 2024-04-28T12:50:46.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.227+0000 7f6a69800640 1 -- 172.21.15.84:0/777607884 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a641a2bb0 con 0x7f6a641028c0 2024-04-28T12:50:46.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.227+0000 7f6a69800640 1 -- 172.21.15.84:0/777607884 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a641a3080 con 0x7f6a641028c0 2024-04-28T12:50:46.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.228+0000 7f6a69800640 1 -- 172.21.15.84:0/777607884 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a641a3250 con 0x7f6a641028c0 2024-04-28T12:50:46.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.231+0000 7f6a60c00640 1 -- 172.21.15.84:0/777607884 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6a54004000 con 0x7f6a641028c0 2024-04-28T12:50:46.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.232+0000 7f6a60c00640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a480a3c40 0x7f6a480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:46.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.232+0000 7f6a60c00640 1 -- 172.21.15.84:0/777607884 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6a540d0fe0 con 0x7f6a641028c0 2024-04-28T12:50:46.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.232+0000 7f6a63400640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a480a3c40 0x7f6a480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:46.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.233+0000 7f6a63400640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a480a3c40 0x7f6a480a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6a50031500 tx=0x7f6a50005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:46.231 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.234+0000 7f6a60c00640 1 -- 172.21.15.84:0/777607884 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6a5409d4e0 con 0x7f6a641028c0 2024-04-28T12:50:46.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:45 smithi161 ceph-mon[62387]: pgmap v392: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:46.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.435+0000 7f6a69800640 1 -- 172.21.15.84:0/777607884 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6a641a3420 con 0x7f6a641028c0 2024-04-28T12:50:46.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.436+0000 7f6a60c00640 1 -- 172.21.15.84:0/777607884 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f6a5409cde0 con 0x7f6a641028c0 2024-04-28T12:50:46.434 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-04-28T12:50:46.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.439+0000 7f6a69800640 1 -- 172.21.15.84:0/777607884 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a480a3c40 msgr2=0x7f6a480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:46.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.439+0000 7f6a69800640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a480a3c40 0x7f6a480a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6a50031500 tx=0x7f6a50005a20 comp rx=0 tx=0).stop 2024-04-28T12:50:46.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.439+0000 7f6a69800640 1 -- 172.21.15.84:0/777607884 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a641028c0 msgr2=0x7f6a6419ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:46.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.439+0000 7f6a69800640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a641028c0 0x7f6a6419ac80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6a5400e9c0 tx=0x7f6a5400ee80 comp rx=0 tx=0).stop 2024-04-28T12:50:46.436 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.439+0000 7f6a69800640 1 -- 172.21.15.84:0/777607884 shutdown_connections 2024-04-28T12:50:46.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.439+0000 7f6a69800640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6a480a3c40 0x7f6a480a6100 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.439+0000 7f6a69800640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a64104470 0x7f6a641a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.439+0000 7f6a69800640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a64103ab0 0x7f6a6419b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.439+0000 7f6a69800640 1 --2- 172.21.15.84:0/777607884 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6a641028c0 0x7f6a6419ac80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.439+0000 7f6a69800640 1 -- 172.21.15.84:0/777607884 >> 172.21.15.84:0/777607884 conn(0x7f6a640fde30 msgr2=0x7f6a64074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:46.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.439+0000 7f6a69800640 1 -- 172.21.15.84:0/777607884 shutdown_connections 2024-04-28T12:50:46.437 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.439+0000 7f6a69800640 1 -- 172.21.15.84:0/777607884 wait complete. 2024-04-28T12:50:46.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T12:50:46.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.667+0000 7fd51a600640 1 -- 172.21.15.84:0/1080510085 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd514104530 msgr2=0x7fd51410adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:46.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.667+0000 7fd51a600640 1 --2- 172.21.15.84:0/1080510085 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd514104530 0x7fd51410adc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd50c009be0 tx=0x7fd50c02a6c0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.668+0000 7fd51a600640 1 -- 172.21.15.84:0/1080510085 shutdown_connections 2024-04-28T12:50:46.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.668+0000 7fd51a600640 1 --2- 172.21.15.84:0/1080510085 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd514104530 0x7fd51410adc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.668+0000 7fd51a600640 1 --2- 172.21.15.84:0/1080510085 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd514103b70 0x7fd514103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.668+0000 7fd51a600640 1 --2- 172.21.15.84:0/1080510085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd514102980 0x7fd514102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.668+0000 7fd51a600640 1 -- 172.21.15.84:0/1080510085 >> 172.21.15.84:0/1080510085 conn(0x7fd5140fdf10 msgr2=0x7fd514100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:46.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.668+0000 7fd51a600640 1 -- 172.21.15.84:0/1080510085 shutdown_connections 2024-04-28T12:50:46.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.668+0000 7fd51a600640 1 -- 172.21.15.84:0/1080510085 wait complete. 2024-04-28T12:50:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.669+0000 7fd51a600640 1 Processor -- start 2024-04-28T12:50:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.669+0000 7fd51a600640 1 -- start start 2024-04-28T12:50:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.669+0000 7fd51a600640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd514102980 0x7fd5140766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.669+0000 7fd51a600640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd514103b70 0x7fd514074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.669+0000 7fd51a600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd514104530 0x7fd514075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.669+0000 7fd51a600640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd51410e490 con 0x7fd514102980 2024-04-28T12:50:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.669+0000 7fd51a600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd51410e310 con 0x7fd514104530 2024-04-28T12:50:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.669+0000 7fd51a600640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd51410e610 con 0x7fd514103b70 2024-04-28T12:50:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.669+0000 7fd513400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd514102980 0x7fd5140766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.669+0000 7fd513400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd514102980 0x7fd5140766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55490/0 (socket says 172.21.15.84:55490) 2024-04-28T12:50:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.669+0000 7fd513400640 1 -- 172.21.15.84:0/763118016 learned_addr learned my addr 172.21.15.84:0/763118016 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.670+0000 7fd50be00640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd514103b70 0x7fd514074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.670+0000 7fd513e00640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd514104530 0x7fd514075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.670+0000 7fd513400640 1 -- 172.21.15.84:0/763118016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd514103b70 msgr2=0x7fd514074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.670+0000 7fd513400640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd514103b70 0x7fd514074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.670+0000 7fd513400640 1 -- 172.21.15.84:0/763118016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd514104530 msgr2=0x7fd514075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.670+0000 7fd513400640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd514104530 0x7fd514075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.670+0000 7fd513400640 1 -- 172.21.15.84:0/763118016 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd514075af0 con 0x7fd514102980 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.670+0000 7fd513e00640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd514104530 0x7fd514075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.670+0000 7fd513400640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd514102980 0x7fd5140766e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd50400e9a0 tx=0x7fd50400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.670+0000 7fd510c00640 1 -- 172.21.15.84:0/763118016 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50400cd80 con 0x7fd514102980 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.671+0000 7fd510c00640 1 -- 172.21.15.84:0/763118016 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5040108e0 con 0x7fd514102980 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.671+0000 7fd51a600640 1 -- 172.21.15.84:0/763118016 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5141a2c60 con 0x7fd514102980 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.671+0000 7fd510c00640 1 -- 172.21.15.84:0/763118016 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd504018960 con 0x7fd514102980 2024-04-28T12:50:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.671+0000 7fd51a600640 1 -- 172.21.15.84:0/763118016 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5141a31b0 con 0x7fd514102980 2024-04-28T12:50:46.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.672+0000 7fd51a600640 1 -- 172.21.15.84:0/763118016 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd514109ca0 con 0x7fd514102980 2024-04-28T12:50:46.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.674+0000 7fd510c00640 1 -- 172.21.15.84:0/763118016 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd504010a90 con 0x7fd514102980 2024-04-28T12:50:46.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.675+0000 7fd510c00640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd4f80a3c40 0x7fd4f80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:46.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.675+0000 7fd510c00640 1 -- 172.21.15.84:0/763118016 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fd5040d0d50 con 0x7fd514102980 2024-04-28T12:50:46.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.675+0000 7fd50be00640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd4f80a3c40 0x7fd4f80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:46.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.676+0000 7fd50be00640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd4f80a3c40 0x7fd4f80a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd514076450 tx=0x7fd4fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:46.676 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.679+0000 7fd510c00640 1 -- 172.21.15.84:0/763118016 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd504054060 con 0x7fd514102980 2024-04-28T12:50:46.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.881+0000 7fd51a600640 1 -- 172.21.15.84:0/763118016 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd514063530 con 0x7fd514102980 2024-04-28T12:50:46.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.882+0000 7fd510c00640 1 -- 172.21.15.84:0/763118016 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fd50409d1f0 con 0x7fd514102980 2024-04-28T12:50:46.880 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T12:50:46.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.885+0000 7fd51a600640 1 -- 172.21.15.84:0/763118016 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd4f80a3c40 msgr2=0x7fd4f80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:46.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.885+0000 7fd51a600640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd4f80a3c40 0x7fd4f80a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd514076450 tx=0x7fd4fc005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:46.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.885+0000 7fd51a600640 1 -- 172.21.15.84:0/763118016 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd514102980 msgr2=0x7fd5140766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:46.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.885+0000 7fd51a600640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd514102980 0x7fd5140766e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd50400e9a0 tx=0x7fd50400ee60 comp rx=0 tx=0).stop 2024-04-28T12:50:46.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.885+0000 7fd51a600640 1 -- 172.21.15.84:0/763118016 shutdown_connections 2024-04-28T12:50:46.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.885+0000 7fd51a600640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd4f80a3c40 0x7fd4f80a6100 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.885+0000 7fd51a600640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd514104530 0x7fd514075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.885+0000 7fd51a600640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd514103b70 0x7fd514074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.885+0000 7fd51a600640 1 --2- 172.21.15.84:0/763118016 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd514102980 0x7fd5140766e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:46.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.885+0000 7fd51a600640 1 -- 172.21.15.84:0/763118016 >> 172.21.15.84:0/763118016 conn(0x7fd5140fdf10 msgr2=0x7fd514106e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:46.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.885+0000 7fd51a600640 1 -- 172.21.15.84:0/763118016 shutdown_connections 2024-04-28T12:50:46.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:46.886+0000 7fd51a600640 1 -- 172.21.15.84:0/763118016 wait complete. 2024-04-28T12:50:46.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T12:50:47.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:46 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4114129520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:47.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:46 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/777607884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:47.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.112+0000 7fc649000640 1 -- 172.21.15.84:0/3423616731 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc64410a5e0 msgr2=0x7fc64410ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.112+0000 7fc649000640 1 --2- 172.21.15.84:0/3423616731 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc64410a5e0 0x7fc64410ca80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc63c009ca0 tx=0x7fc63c02a950 comp rx=0 tx=0).stop 2024-04-28T12:50:47.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.112+0000 7fc649000640 1 -- 172.21.15.84:0/3423616731 shutdown_connections 2024-04-28T12:50:47.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.112+0000 7fc649000640 1 --2- 172.21.15.84:0/3423616731 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc64410a5e0 0x7fc64410ca80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.112+0000 7fc649000640 1 --2- 172.21.15.84:0/3423616731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc644075330 0x7fc644075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.112+0000 7fc649000640 1 --2- 172.21.15.84:0/3423616731 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc644076910 0x7fc644074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.112+0000 7fc649000640 1 -- 172.21.15.84:0/3423616731 >> 172.21.15.84:0/3423616731 conn(0x7fc6440fdcc0 msgr2=0x7fc644100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:47.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.112+0000 7fc649000640 1 -- 172.21.15.84:0/3423616731 shutdown_connections 2024-04-28T12:50:47.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.112+0000 7fc649000640 1 -- 172.21.15.84:0/3423616731 wait complete. 2024-04-28T12:50:47.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.113+0000 7fc649000640 1 Processor -- start 2024-04-28T12:50:47.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.113+0000 7fc649000640 1 -- start start 2024-04-28T12:50:47.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.113+0000 7fc649000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc644075330 0x7fc64419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:47.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.113+0000 7fc649000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc644076910 0x7fc64419b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:47.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.113+0000 7fc649000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc64410a5e0 0x7fc6441a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:47.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.113+0000 7fc649000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc644110480 con 0x7fc64410a5e0 2024-04-28T12:50:47.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.113+0000 7fc649000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc644110300 con 0x7fc644075330 2024-04-28T12:50:47.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.113+0000 7fc649000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc644110600 con 0x7fc644076910 2024-04-28T12:50:47.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.114+0000 7fc643e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc64410a5e0 0x7fc6441a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:47.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.114+0000 7fc643400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc644075330 0x7fc64419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:47.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.114+0000 7fc643e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc64410a5e0 0x7fc6441a20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55508/0 (socket says 172.21.15.84:55508) 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.114+0000 7fc643e00640 1 -- 172.21.15.84:0/3791342806 learned_addr learned my addr 172.21.15.84:0/3791342806 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.114+0000 7fc642a00640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc644076910 0x7fc64419b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.114+0000 7fc643e00640 1 -- 172.21.15.84:0/3791342806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc644076910 msgr2=0x7fc64419b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.114+0000 7fc643e00640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc644076910 0x7fc64419b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.114+0000 7fc643e00640 1 -- 172.21.15.84:0/3791342806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc644075330 msgr2=0x7fc64419ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.114+0000 7fc643e00640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc644075330 0x7fc64419ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.114+0000 7fc643e00640 1 -- 172.21.15.84:0/3791342806 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6441a2730 con 0x7fc64410a5e0 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.114+0000 7fc643e00640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc64410a5e0 0x7fc6441a20c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc63c006010 tx=0x7fc63c02fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.115+0000 7fc63be00640 1 -- 172.21.15.84:0/3791342806 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c034040 con 0x7fc64410a5e0 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.115+0000 7fc63be00640 1 -- 172.21.15.84:0/3791342806 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc63c0308f0 con 0x7fc64410a5e0 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.115+0000 7fc63be00640 1 -- 172.21.15.84:0/3791342806 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c039450 con 0x7fc64410a5e0 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.115+0000 7fc649000640 1 -- 172.21.15.84:0/3791342806 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6441a29c0 con 0x7fc64410a5e0 2024-04-28T12:50:47.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.115+0000 7fc649000640 1 -- 172.21.15.84:0/3791342806 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6441a2e60 con 0x7fc64410a5e0 2024-04-28T12:50:47.114 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.116+0000 7fc649000640 1 -- 172.21.15.84:0/3791342806 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6441093a0 con 0x7fc64410a5e0 2024-04-28T12:50:47.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.118+0000 7fc63be00640 1 -- 172.21.15.84:0/3791342806 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc63c037020 con 0x7fc64410a5e0 2024-04-28T12:50:47.116 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.119+0000 7fc63be00640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc6280a3c40 0x7fc6280a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:47.117 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.120+0000 7fc63be00640 1 -- 172.21.15.84:0/3791342806 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fc63c0f1860 con 0x7fc64410a5e0 2024-04-28T12:50:47.117 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.120+0000 7fc643400640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc6280a3c40 0x7fc6280a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:47.117 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.121+0000 7fc643400640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc6280a3c40 0x7fc6280a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc634009bd0 tx=0x7fc634005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:47.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.124+0000 7fc63be00640 1 -- 172.21.15.84:0/3791342806 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc63c0bdea0 con 0x7fc64410a5e0 2024-04-28T12:50:47.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:46 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4114129520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:47.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:46 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/777607884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:47.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:46 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4114129520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:47.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:46 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/777607884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:47.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.326+0000 7fc649000640 1 -- 172.21.15.84:0/3791342806 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc64402dcd0 con 0x7fc64410a5e0 2024-04-28T12:50:47.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.327+0000 7fc63be00640 1 -- 172.21.15.84:0/3791342806 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fc63c0bd7a0 con 0x7fc64410a5e0 2024-04-28T12:50:47.324 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 64 2024-04-28T12:50:47.326 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.329+0000 7fc649000640 1 -- 172.21.15.84:0/3791342806 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc6280a3c40 msgr2=0x7fc6280a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.329+0000 7fc649000640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc6280a3c40 0x7fc6280a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc634009bd0 tx=0x7fc634005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.329+0000 7fc649000640 1 -- 172.21.15.84:0/3791342806 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc64410a5e0 msgr2=0x7fc6441a20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.329+0000 7fc649000640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc64410a5e0 0x7fc6441a20c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc63c006010 tx=0x7fc63c02fc20 comp rx=0 tx=0).stop 2024-04-28T12:50:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.329+0000 7fc649000640 1 -- 172.21.15.84:0/3791342806 shutdown_connections 2024-04-28T12:50:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.329+0000 7fc649000640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc6280a3c40 0x7fc6280a6100 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.329+0000 7fc649000640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc64410a5e0 0x7fc6441a20c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.329+0000 7fc649000640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc644076910 0x7fc64419b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.329+0000 7fc649000640 1 --2- 172.21.15.84:0/3791342806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc644075330 0x7fc64419ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.329+0000 7fc649000640 1 -- 172.21.15.84:0/3791342806 >> 172.21.15.84:0/3791342806 conn(0x7fc6440fdcc0 msgr2=0x7fc644106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.329+0000 7fc649000640 1 -- 172.21.15.84:0/3791342806 shutdown_connections 2024-04-28T12:50:47.327 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.329+0000 7fc649000640 1 -- 172.21.15.84:0/3791342806 wait complete. 2024-04-28T12:50:47.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T12:50:47.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.549+0000 7f3b16800640 1 -- 172.21.15.84:0/1709329190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b10104a70 msgr2=0x7f3b10104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.549+0000 7f3b16800640 1 --2- 172.21.15.84:0/1709329190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b10104a70 0x7f3b10104e90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3b08009900 tx=0x7f3b0802a3f0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.549+0000 7f3b16800640 1 -- 172.21.15.84:0/1709329190 shutdown_connections 2024-04-28T12:50:47.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.549+0000 7f3b16800640 1 --2- 172.21.15.84:0/1709329190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b10106620 0x7f3b1010ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.549+0000 7f3b16800640 1 --2- 172.21.15.84:0/1709329190 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b10105c60 0x7f3b101060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.549+0000 7f3b16800640 1 --2- 172.21.15.84:0/1709329190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b10104a70 0x7f3b10104e90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.549+0000 7f3b16800640 1 -- 172.21.15.84:0/1709329190 >> 172.21.15.84:0/1709329190 conn(0x7f3b100fffe0 msgr2=0x7f3b10102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:47.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.550+0000 7f3b16800640 1 -- 172.21.15.84:0/1709329190 shutdown_connections 2024-04-28T12:50:47.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.550+0000 7f3b16800640 1 -- 172.21.15.84:0/1709329190 wait complete. 2024-04-28T12:50:47.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.550+0000 7f3b16800640 1 Processor -- start 2024-04-28T12:50:47.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.551+0000 7f3b16800640 1 -- start start 2024-04-28T12:50:47.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.551+0000 7f3b16800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b10104a70 0x7f3b1019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:47.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.551+0000 7f3b16800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b10105c60 0x7f3b1019b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:47.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.551+0000 7f3b16800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b10106620 0x7f3b101a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:47.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.551+0000 7f3b16800640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f3b101105c0 con 0x7f3b10105c60 2024-04-28T12:50:47.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.551+0000 7f3b16800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3b10110440 con 0x7f3b10104a70 2024-04-28T12:50:47.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.551+0000 7f3b16800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f3b10110740 con 0x7f3b10106620 2024-04-28T12:50:47.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.551+0000 7f3b14a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b10105c60 0x7f3b1019b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:47.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.551+0000 7f3b14a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b10105c60 0x7f3b1019b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55530/0 (socket says 172.21.15.84:55530) 2024-04-28T12:50:47.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.551+0000 7f3b15400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b10104a70 0x7f3b1019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:47.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.551+0000 7f3b14a00640 1 -- 172.21.15.84:0/1747392931 learned_addr learned my addr 172.21.15.84:0/1747392931 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:47.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.551+0000 7f3b15e00640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b10106620 0x7f3b101a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:47.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.552+0000 7f3b15400640 1 -- 172.21.15.84:0/1747392931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b10106620 msgr2=0x7f3b101a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.552+0000 7f3b15400640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b10106620 0x7f3b101a2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.552+0000 7f3b15400640 1 -- 172.21.15.84:0/1747392931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b10105c60 msgr2=0x7f3b1019b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.552+0000 7f3b15400640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b10105c60 0x7f3b1019b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.552+0000 7f3b15400640 1 -- 172.21.15.84:0/1747392931 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b101a28e0 con 0x7f3b10104a70 2024-04-28T12:50:47.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.552+0000 7f3b14a00640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b10105c60 0x7f3b1019b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:47.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.552+0000 7f3b15400640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b10104a70 0x7f3b1019aca0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3b0802a910 tx=0x7f3b08003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:47.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.552+0000 7f3b0e000640 1 -- 172.21.15.84:0/1747392931 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0803f070 con 0x7f3b10104a70 2024-04-28T12:50:47.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.552+0000 7f3b16800640 1 -- 172.21.15.84:0/1747392931 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b101a2b70 con 0x7f3b10104a70 2024-04-28T12:50:47.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.553+0000 7f3b0e000640 1 -- 172.21.15.84:0/1747392931 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b08031c50 con 0x7f3b10104a70 2024-04-28T12:50:47.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.553+0000 7f3b0e000640 1 -- 172.21.15.84:0/1747392931 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b08035030 con 0x7f3b10104a70 2024-04-28T12:50:47.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.553+0000 7f3b16800640 1 -- 172.21.15.84:0/1747392931 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b101a3080 con 0x7f3b10104a70 2024-04-28T12:50:47.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.554+0000 7f3b16800640 1 -- 172.21.15.84:0/1747392931 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b1010be10 con 0x7f3b10104a70 2024-04-28T12:50:47.553 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.556+0000 7f3b0e000640 1 -- 172.21.15.84:0/1747392931 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3b08042070 con 0x7f3b10104a70 2024-04-28T12:50:47.554 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.557+0000 7f3b0e000640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3af40a3c40 0x7f3af40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:47.554 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.557+0000 7f3b0e000640 1 -- 172.21.15.84:0/1747392931 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f3b080f19c0 con 0x7f3b10104a70 2024-04-28T12:50:47.554 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.557+0000 7f3b14a00640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3af40a3c40 0x7f3af40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:47.555 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.558+0000 7f3b14a00640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3af40a3c40 0x7f3af40a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3b1019c1a0 tx=0x7f3afc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:47.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.560+0000 7f3b0e000640 1 -- 172.21.15.84:0/1747392931 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3b080bd810 con 0x7f3b10104a70 2024-04-28T12:50:47.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.769+0000 7f3b16800640 1 -- 172.21.15.84:0/1747392931 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3b1002dc60 con 0x7f3b10104a70 2024-04-28T12:50:47.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.770+0000 7f3b0e000640 1 -- 172.21.15.84:0/1747392931 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f3b080c1b10 con 0x7f3b10104a70 2024-04-28T12:50:47.767 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T12:50:47.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.773+0000 7f3b16800640 1 -- 172.21.15.84:0/1747392931 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3af40a3c40 msgr2=0x7f3af40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.773+0000 7f3b16800640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3af40a3c40 0x7f3af40a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3b1019c1a0 tx=0x7f3afc009500 comp rx=0 tx=0).stop 2024-04-28T12:50:47.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.773+0000 7f3b16800640 1 -- 172.21.15.84:0/1747392931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b10104a70 msgr2=0x7f3b1019aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.773+0000 7f3b16800640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b10104a70 0x7f3b1019aca0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3b0802a910 tx=0x7f3b08003c10 comp rx=0 tx=0).stop 2024-04-28T12:50:47.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.773+0000 7f3b16800640 1 -- 172.21.15.84:0/1747392931 shutdown_connections 2024-04-28T12:50:47.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.773+0000 7f3b16800640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f3af40a3c40 0x7f3af40a6100 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.773+0000 7f3b16800640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b10106620 0x7f3b101a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.773+0000 7f3b16800640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b10105c60 0x7f3b1019b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.773+0000 7f3b16800640 1 --2- 172.21.15.84:0/1747392931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b10104a70 0x7f3b1019aca0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.773+0000 7f3b16800640 1 -- 172.21.15.84:0/1747392931 >> 172.21.15.84:0/1747392931 conn(0x7f3b100fffe0 msgr2=0x7f3b10108fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:47.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.773+0000 7f3b16800640 1 -- 172.21.15.84:0/1747392931 shutdown_connections 2024-04-28T12:50:47.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.773+0000 7f3b16800640 1 -- 172.21.15.84:0/1747392931 wait complete. 2024-04-28T12:50:47.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:50:47.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.992+0000 7f37a9200640 1 -- 172.21.15.84:0/2593692880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37a4105b20 msgr2=0x7f37a4107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.992+0000 7f37a9200640 1 --2- 172.21.15.84:0/2593692880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37a4105b20 0x7f37a4107f10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f378c009900 tx=0x7f378c02a3f0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.992+0000 7f37a9200640 1 -- 172.21.15.84:0/2593692880 shutdown_connections 2024-04-28T12:50:47.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.992+0000 7f37a9200640 1 --2- 172.21.15.84:0/2593692880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37a4108450 0x7f37a410a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.992+0000 7f37a9200640 1 --2- 172.21.15.84:0/2593692880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37a4105b20 0x7f37a4107f10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.992+0000 7f37a9200640 1 --2- 172.21.15.84:0/2593692880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37a41031f0 0x7f37a41055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.992+0000 7f37a9200640 1 -- 172.21.15.84:0/2593692880 >> 172.21.15.84:0/2593692880 conn(0x7f37a40fcbc0 msgr2=0x7f37a40ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:47.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.992+0000 7f37a9200640 1 -- 172.21.15.84:0/2593692880 shutdown_connections 2024-04-28T12:50:47.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.992+0000 7f37a9200640 1 -- 172.21.15.84:0/2593692880 wait complete. 2024-04-28T12:50:47.992 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.993+0000 7f37a9200640 1 Processor -- start 2024-04-28T12:50:47.992 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.993+0000 7f37a9200640 1 -- start start 2024-04-28T12:50:47.992 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.993+0000 7f37a9200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37a41031f0 0x7f37a4102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:47.992 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.993+0000 7f37a9200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37a4105b20 0x7f37a41011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:47.992 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.993+0000 7f37a9200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37a4108450 0x7f37a41016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:47.992 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.993+0000 7f37a9200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f37a410e2e0 con 0x7f37a4105b20 2024-04-28T12:50:47.992 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.993+0000 7f37a9200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f37a410e160 con 0x7f37a41031f0 2024-04-28T12:50:47.992 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.993+0000 7f37a9200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f37a410e460 con 0x7f37a4108450 2024-04-28T12:50:47.992 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.993+0000 7f379aa00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37a4105b20 0x7f37a41011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.993+0000 7f379aa00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37a4105b20 0x7f37a41011a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55550/0 (socket says 172.21.15.84:55550) 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.994+0000 7f379aa00640 1 -- 172.21.15.84:0/4199844750 learned_addr learned my addr 172.21.15.84:0/4199844750 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.994+0000 7f37a3400640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37a41031f0 0x7f37a4102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.994+0000 7f37a3e00640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37a4108450 0x7f37a41016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.994+0000 7f379aa00640 1 -- 172.21.15.84:0/4199844750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37a4108450 msgr2=0x7f37a41016e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.994+0000 7f379aa00640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37a4108450 0x7f37a41016e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.994+0000 7f379aa00640 1 -- 172.21.15.84:0/4199844750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37a41031f0 msgr2=0x7f37a4102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.994+0000 7f379aa00640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37a41031f0 0x7f37a4102b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.994+0000 7f379aa00640 1 -- 172.21.15.84:0/4199844750 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a4101f70 con 0x7f37a4105b20 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.994+0000 7f379aa00640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37a4105b20 0x7f37a41011a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f378c031500 tx=0x7f378c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.994+0000 7f37a0c00640 1 -- 172.21.15.84:0/4199844750 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378c03f070 con 0x7f37a4105b20 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.995+0000 7f37a9200640 1 -- 172.21.15.84:0/4199844750 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37a41a2a60 con 0x7f37a4105b20 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.995+0000 7f37a0c00640 1 -- 172.21.15.84:0/4199844750 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f378c031c50 con 0x7f37a4105b20 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.995+0000 7f37a0c00640 1 -- 172.21.15.84:0/4199844750 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378c035030 con 0x7f37a4105b20 2024-04-28T12:50:47.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.995+0000 7f37a9200640 1 -- 172.21.15.84:0/4199844750 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37a41a3000 con 0x7f37a4105b20 2024-04-28T12:50:47.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.996+0000 7f37a9200640 1 -- 172.21.15.84:0/4199844750 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37a40fe6b0 con 0x7f37a4105b20 2024-04-28T12:50:47.995 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.998+0000 7f37a0c00640 1 -- 172.21.15.84:0/4199844750 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f378c042070 con 0x7f37a4105b20 2024-04-28T12:50:47.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.999+0000 7f37a0c00640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f37880a3c40 0x7f37880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:47.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.999+0000 7f37a0c00640 1 -- 172.21.15.84:0/4199844750 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f378c0f0ff0 con 0x7f37a4105b20 2024-04-28T12:50:47.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:47.999+0000 7f37a3400640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f37880a3c40 0x7f37880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:47.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.000+0000 7f37a3400640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f37880a3c40 0x7f37880a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f37940095f0 tx=0x7f3794005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:48.000 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.003+0000 7f37a0c00640 1 -- 172.21.15.84:0/4199844750 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f378c0bce80 con 0x7f37a4105b20 2024-04-28T12:50:48.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:47 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/763118016' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:47 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3791342806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:47 smithi179 ceph-mon[63029]: pgmap v393: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:47 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/763118016' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:47 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3791342806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:47 smithi084 ceph-mon[28081]: pgmap v393: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:48.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.245+0000 7f37a9200640 1 -- 172.21.15.84:0/4199844750 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37a402dca0 con 0x7f37a4105b20 2024-04-28T12:50:48.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:47 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/763118016' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:48.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:47 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3791342806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:48.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:47 smithi161 ceph-mon[62387]: pgmap v393: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:48.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.247+0000 7f37a0c00640 1 -- 172.21.15.84:0/4199844750 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11884 (secure 0 0 0) 0x7f378c0c1180 con 0x7f37a4105b20 2024-04-28T12:50:48.245 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:50:48.245 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6841/1196924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":1196924806},{"type":"v1","addr":"172.21.15.84:6841","nonce":1196924806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:34.334879+0000","modified":"2024-04-28T12:49:08.738419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24562,"mds_2":24545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6839/3812720642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":3812720642},{"type":"v1","addr":"172.21.15.161:6839","nonce":3812720642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24545,24562,24602]},"id":1}]} 2024-04-28T12:50:48.245 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 24 2024-04-28T12:50:48.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.250+0000 7f37a9200640 1 -- 172.21.15.84:0/4199844750 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f37880a3c40 msgr2=0x7f37880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:48.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.250+0000 7f37a9200640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f37880a3c40 0x7f37880a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f37940095f0 tx=0x7f3794005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:48.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.250+0000 7f37a9200640 1 -- 172.21.15.84:0/4199844750 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37a4105b20 msgr2=0x7f37a41011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:48.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.250+0000 7f37a9200640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37a4105b20 0x7f37a41011a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f378c031500 tx=0x7f378c003c10 comp rx=0 tx=0).stop 2024-04-28T12:50:48.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.250+0000 7f37a9200640 1 -- 172.21.15.84:0/4199844750 shutdown_connections 2024-04-28T12:50:48.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.250+0000 7f37a9200640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f37880a3c40 0x7f37880a6100 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.250+0000 7f37a9200640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37a4108450 0x7f37a41016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.250+0000 7f37a9200640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f37a4105b20 0x7f37a41011a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.250+0000 7f37a9200640 1 --2- 172.21.15.84:0/4199844750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37a41031f0 0x7f37a4102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.250+0000 7f37a9200640 1 -- 172.21.15.84:0/4199844750 >> 172.21.15.84:0/4199844750 conn(0x7f37a40fcbc0 msgr2=0x7f37a4103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:48.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.251+0000 7f37a9200640 1 -- 172.21.15.84:0/4199844750 shutdown_connections 2024-04-28T12:50:48.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.251+0000 7f37a9200640 1 -- 172.21.15.84:0/4199844750 wait complete. 2024-04-28T12:50:48.266 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:50:48.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.474+0000 7f6c42000640 1 -- 172.21.15.84:0/1235368696 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c3c108d90 msgr2=0x7f6c3c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:48.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.474+0000 7f6c42000640 1 --2- 172.21.15.84:0/1235368696 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c3c108d90 0x7f6c3c1091b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6c30009900 tx=0x7f6c3002a410 comp rx=0 tx=0).stop 2024-04-28T12:50:48.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.474+0000 7f6c42000640 1 -- 172.21.15.84:0/1235368696 shutdown_connections 2024-04-28T12:50:48.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.474+0000 7f6c42000640 1 --2- 172.21.15.84:0/1235368696 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c3c10a940 0x7f6c3c1111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.474+0000 7f6c42000640 1 --2- 172.21.15.84:0/1235368696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c3c109f80 0x7f6c3c10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.474+0000 7f6c42000640 1 --2- 172.21.15.84:0/1235368696 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c3c108d90 0x7f6c3c1091b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.474+0000 7f6c42000640 1 -- 172.21.15.84:0/1235368696 >> 172.21.15.84:0/1235368696 conn(0x7f6c3c077d30 msgr2=0x7f6c3c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:48.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.474+0000 7f6c42000640 1 -- 172.21.15.84:0/1235368696 shutdown_connections 2024-04-28T12:50:48.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.474+0000 7f6c42000640 1 -- 172.21.15.84:0/1235368696 wait complete. 2024-04-28T12:50:48.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.475+0000 7f6c42000640 1 Processor -- start 2024-04-28T12:50:48.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.475+0000 7f6c42000640 1 -- start start 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c42000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c3c108d90 0x7f6c3c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c42000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c3c109f80 0x7f6c3c19f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c42000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c3c10a940 0x7f6c3c1a6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c42000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c114910 con 0x7f6c3c108d90 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c42000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c114790 con 0x7f6c3c109f80 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c42000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c114a90 con 0x7f6c3c10a940 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c40c00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c3c108d90 0x7f6c3c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c40c00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c3c108d90 0x7f6c3c19efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55566/0 (socket says 172.21.15.84:55566) 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c40c00640 1 -- 172.21.15.84:0/336704994 learned_addr learned my addr 172.21.15.84:0/336704994 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c3be00640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c3c109f80 0x7f6c3c19f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c41600640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c3c10a940 0x7f6c3c1a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c40c00640 1 -- 172.21.15.84:0/336704994 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c3c10a940 msgr2=0x7f6c3c1a6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c40c00640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c3c10a940 0x7f6c3c1a6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c40c00640 1 -- 172.21.15.84:0/336704994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c3c109f80 msgr2=0x7f6c3c19f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:48.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c40c00640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c3c109f80 0x7f6c3c19f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c40c00640 1 -- 172.21.15.84:0/336704994 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3c1a6bc0 con 0x7f6c3c108d90 2024-04-28T12:50:48.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.476+0000 7f6c41600640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c3c10a940 0x7f6c3c1a6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:48.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.477+0000 7f6c3be00640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c3c109f80 0x7f6c3c19f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:48.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.477+0000 7f6c40c00640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c3c108d90 0x7f6c3c19efa0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6c30031500 tx=0x7f6c30003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:48.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.477+0000 7f6c39600640 1 -- 172.21.15.84:0/336704994 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3003f070 con 0x7f6c3c108d90 2024-04-28T12:50:48.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.477+0000 7f6c39600640 1 -- 172.21.15.84:0/336704994 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c30031c50 con 0x7f6c3c108d90 2024-04-28T12:50:48.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.477+0000 7f6c42000640 1 -- 172.21.15.84:0/336704994 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3c1a6e50 con 0x7f6c3c108d90 2024-04-28T12:50:48.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.477+0000 7f6c39600640 1 -- 172.21.15.84:0/336704994 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c30035030 con 0x7f6c3c108d90 2024-04-28T12:50:48.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.477+0000 7f6c42000640 1 -- 172.21.15.84:0/336704994 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c3c1a72b0 con 0x7f6c3c108d90 2024-04-28T12:50:48.476 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.478+0000 7f6c42000640 1 -- 172.21.15.84:0/336704994 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c3c110180 con 0x7f6c3c108d90 2024-04-28T12:50:48.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.481+0000 7f6c39600640 1 -- 172.21.15.84:0/336704994 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6c30042070 con 0x7f6c3c108d90 2024-04-28T12:50:48.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.482+0000 7f6c39600640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6c240a3cc0 0x7f6c240a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:48.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.482+0000 7f6c39600640 1 -- 172.21.15.84:0/336704994 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6c300f1790 con 0x7f6c3c108d90 2024-04-28T12:50:48.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.482+0000 7f6c3be00640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6c240a3cc0 0x7f6c240a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:48.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.483+0000 7f6c3be00640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6c240a3cc0 0x7f6c240a6180 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6c280095f0 tx=0x7f6c28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:48.481 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.485+0000 7f6c39600640 1 -- 172.21.15.84:0/336704994 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6c300bdc30 con 0x7f6c3c108d90 2024-04-28T12:50:48.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.688+0000 7f6c42000640 1 -- 172.21.15.84:0/336704994 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c3c02dca0 con 0x7f6c3c108d90 2024-04-28T12:50:48.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.691+0000 7f6c39600640 1 -- 172.21.15.84:0/336704994 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24634 (secure 0 0 0) 0x7f6c300bd530 con 0x7f6c3c108d90 2024-04-28T12:50:48.688 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:50:48.688 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":94,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:50:30.848698+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.8","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:50:48.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.693+0000 7f6c42000640 1 -- 172.21.15.84:0/336704994 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6c240a3cc0 msgr2=0x7f6c240a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:48.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.693+0000 7f6c42000640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6c240a3cc0 0x7f6c240a6180 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6c280095f0 tx=0x7f6c28005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:48.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.693+0000 7f6c42000640 1 -- 172.21.15.84:0/336704994 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c3c108d90 msgr2=0x7f6c3c19efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:48.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.693+0000 7f6c42000640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c3c108d90 0x7f6c3c19efa0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6c30031500 tx=0x7f6c30003c10 comp rx=0 tx=0).stop 2024-04-28T12:50:48.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.694+0000 7f6c42000640 1 -- 172.21.15.84:0/336704994 shutdown_connections 2024-04-28T12:50:48.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.694+0000 7f6c42000640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6c240a3cc0 0x7f6c240a6180 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.694+0000 7f6c42000640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c3c10a940 0x7f6c3c1a6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.694+0000 7f6c42000640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c3c109f80 0x7f6c3c19f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.694+0000 7f6c42000640 1 --2- 172.21.15.84:0/336704994 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c3c108d90 0x7f6c3c19efa0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.694+0000 7f6c42000640 1 -- 172.21.15.84:0/336704994 >> 172.21.15.84:0/336704994 conn(0x7f6c3c077d30 msgr2=0x7f6c3c10d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:48.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.694+0000 7f6c42000640 1 -- 172.21.15.84:0/336704994 shutdown_connections 2024-04-28T12:50:48.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.694+0000 7f6c42000640 1 -- 172.21.15.84:0/336704994 wait complete. 2024-04-28T12:50:48.711 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:43:15.608080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T12:48:32.037809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T12:48:33.240166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T12:48:37.567238+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T12:50:48.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:50:48.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.922+0000 7fda80c00640 1 -- 172.21.15.84:0/2301542403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda7c10a180 msgr2=0x7fda7c10a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:48.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.922+0000 7fda80c00640 1 --2- 172.21.15.84:0/2301542403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda7c10a180 0x7fda7c10a600 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fda64009900 tx=0x7fda6402a3f0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.923+0000 7fda80c00640 1 -- 172.21.15.84:0/2301542403 shutdown_connections 2024-04-28T12:50:48.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.923+0000 7fda80c00640 1 --2- 172.21.15.84:0/2301542403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda7c10ab40 0x7fda7c1113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.923+0000 7fda80c00640 1 --2- 172.21.15.84:0/2301542403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda7c10a180 0x7fda7c10a600 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.923+0000 7fda80c00640 1 --2- 172.21.15.84:0/2301542403 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda7c108f90 0x7fda7c1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.923+0000 7fda80c00640 1 -- 172.21.15.84:0/2301542403 >> 172.21.15.84:0/2301542403 conn(0x7fda7c077f30 msgr2=0x7fda7c07a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:48.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.923+0000 7fda80c00640 1 -- 172.21.15.84:0/2301542403 shutdown_connections 2024-04-28T12:50:48.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.923+0000 7fda80c00640 1 -- 172.21.15.84:0/2301542403 wait complete. 2024-04-28T12:50:48.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.924+0000 7fda80c00640 1 Processor -- start 2024-04-28T12:50:48.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.924+0000 7fda80c00640 1 -- start start 2024-04-28T12:50:48.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.924+0000 7fda80c00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda7c108f90 0x7fda7c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:48.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.924+0000 7fda80c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda7c10a180 0x7fda7c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:48.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.924+0000 7fda80c00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda7c10ab40 0x7fda7c1a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:48.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.924+0000 7fda80c00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c114b20 con 0x7fda7c10ab40 2024-04-28T12:50:48.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.924+0000 7fda80c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c1149a0 con 0x7fda7c10a180 2024-04-28T12:50:48.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.924+0000 7fda80c00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c114ca0 con 0x7fda7c108f90 2024-04-28T12:50:48.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.925+0000 7fda7a800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda7c10ab40 0x7fda7c1a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:48.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.925+0000 7fda7a800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda7c10ab40 0x7fda7c1a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35264/0 (socket says 172.21.15.84:35264) 2024-04-28T12:50:48.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.925+0000 7fda7a800640 1 -- 172.21.15.84:0/924951447 learned_addr learned my addr 172.21.15.84:0/924951447 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:48.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.925+0000 7fda73e00640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda7c10a180 0x7fda7c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:48.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.925+0000 7fda79e00640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda7c108f90 0x7fda7c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:48.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.925+0000 7fda7a800640 1 -- 172.21.15.84:0/924951447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda7c108f90 msgr2=0x7fda7c19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:48.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.925+0000 7fda7a800640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda7c108f90 0x7fda7c19f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.925+0000 7fda7a800640 1 -- 172.21.15.84:0/924951447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda7c10a180 msgr2=0x7fda7c19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:48.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.925+0000 7fda7a800640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda7c10a180 0x7fda7c19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:48.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.925+0000 7fda7a800640 1 -- 172.21.15.84:0/924951447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda7c1a6dc0 con 0x7fda7c10ab40 2024-04-28T12:50:48.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.925+0000 7fda73e00640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda7c10a180 0x7fda7c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:48.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.925+0000 7fda7a800640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda7c10ab40 0x7fda7c1a6750 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fda7400eb10 tx=0x7fda7400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:48.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.926+0000 7fda72a00640 1 -- 172.21.15.84:0/924951447 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74004b20 con 0x7fda7c10ab40 2024-04-28T12:50:48.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.926+0000 7fda72a00640 1 -- 172.21.15.84:0/924951447 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda74004cc0 con 0x7fda7c10ab40 2024-04-28T12:50:48.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.926+0000 7fda80c00640 1 -- 172.21.15.84:0/924951447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda7c1a70b0 con 0x7fda7c10ab40 2024-04-28T12:50:48.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.926+0000 7fda72a00640 1 -- 172.21.15.84:0/924951447 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74010690 con 0x7fda7c10ab40 2024-04-28T12:50:48.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.926+0000 7fda80c00640 1 -- 172.21.15.84:0/924951447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda7c1a7620 con 0x7fda7c10ab40 2024-04-28T12:50:48.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.927+0000 7fda80c00640 1 -- 172.21.15.84:0/924951447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda7c110330 con 0x7fda7c10ab40 2024-04-28T12:50:48.926 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.929+0000 7fda72a00640 1 -- 172.21.15.84:0/924951447 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fda74005020 con 0x7fda7c10ab40 2024-04-28T12:50:48.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.931+0000 7fda72a00640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fda600a3cc0 0x7fda600a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:48.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.931+0000 7fda72a00640 1 -- 172.21.15.84:0/924951447 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fda740d0e30 con 0x7fda7c10ab40 2024-04-28T12:50:48.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.931+0000 7fda79e00640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fda600a3cc0 0x7fda600a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:48.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.932+0000 7fda79e00640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fda600a3cc0 0x7fda600a6180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fda7c109fe0 tx=0x7fda6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:48.931 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:48.935+0000 7fda72a00640 1 -- 172.21.15.84:0/924951447 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fda7409d2d0 con 0x7fda7c10ab40 2024-04-28T12:50:49.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.134+0000 7fda80c00640 1 -- 172.21.15.84:0/924951447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fda7c063530 con 0x7fda7c10ab40 2024-04-28T12:50:49.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.135+0000 7fda72a00640 1 -- 172.21.15.84:0/924951447 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fda7409cbd0 con 0x7fda7c10ab40 2024-04-28T12:50:49.132 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-04-28T12:50:49.135 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.138+0000 7fda80c00640 1 -- 172.21.15.84:0/924951447 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fda600a3cc0 msgr2=0x7fda600a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:49.135 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.138+0000 7fda80c00640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fda600a3cc0 0x7fda600a6180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fda7c109fe0 tx=0x7fda6c005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:49.135 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.138+0000 7fda80c00640 1 -- 172.21.15.84:0/924951447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda7c10ab40 msgr2=0x7fda7c1a6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:49.135 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.138+0000 7fda80c00640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda7c10ab40 0x7fda7c1a6750 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fda7400eb10 tx=0x7fda7400c2d0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.138+0000 7fda80c00640 1 -- 172.21.15.84:0/924951447 shutdown_connections 2024-04-28T12:50:49.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.138+0000 7fda80c00640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fda600a3cc0 0x7fda600a6180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.138+0000 7fda80c00640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fda7c10ab40 0x7fda7c1a6750 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.138+0000 7fda80c00640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda7c10a180 0x7fda7c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.138+0000 7fda80c00640 1 --2- 172.21.15.84:0/924951447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fda7c108f90 0x7fda7c19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.138+0000 7fda80c00640 1 -- 172.21.15.84:0/924951447 >> 172.21.15.84:0/924951447 conn(0x7fda7c077f30 msgr2=0x7fda7c10d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:49.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.138+0000 7fda80c00640 1 -- 172.21.15.84:0/924951447 shutdown_connections 2024-04-28T12:50:49.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.138+0000 7fda80c00640 1 -- 172.21.15.84:0/924951447 wait complete. 2024-04-28T12:50:49.152 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:48 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1747392931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:50:49.152 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:48 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4199844750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:49.152 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:48 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/336704994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:49.155 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T12:50:49.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:48 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1747392931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:50:49.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:48 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4199844750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:49.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:48 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/336704994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:49.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:48 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1747392931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:50:49.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:48 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4199844750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:49.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:48 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/336704994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:49.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.364+0000 7fd7bc000640 1 -- 172.21.15.84:0/2677227054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b410a180 msgr2=0x7fd7b410a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:49.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.364+0000 7fd7bc000640 1 --2- 172.21.15.84:0/2677227054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b410a180 0x7fd7b410a600 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0009900 tx=0x7fd7a002a410 comp rx=0 tx=0).stop 2024-04-28T12:50:49.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.364+0000 7fd7bc000640 1 -- 172.21.15.84:0/2677227054 shutdown_connections 2024-04-28T12:50:49.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.364+0000 7fd7bc000640 1 --2- 172.21.15.84:0/2677227054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7b410ab40 0x7fd7b41113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.364+0000 7fd7bc000640 1 --2- 172.21.15.84:0/2677227054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b410a180 0x7fd7b410a600 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.364+0000 7fd7bc000640 1 --2- 172.21.15.84:0/2677227054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7b4108f90 0x7fd7b41093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.364+0000 7fd7bc000640 1 -- 172.21.15.84:0/2677227054 >> 172.21.15.84:0/2677227054 conn(0x7fd7b4077f30 msgr2=0x7fd7b407a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:49.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.364+0000 7fd7bc000640 1 -- 172.21.15.84:0/2677227054 shutdown_connections 2024-04-28T12:50:49.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.364+0000 7fd7bc000640 1 -- 172.21.15.84:0/2677227054 wait complete. 2024-04-28T12:50:49.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.365+0000 7fd7bc000640 1 Processor -- start 2024-04-28T12:50:49.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.365+0000 7fd7bc000640 1 -- start start 2024-04-28T12:50:49.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.365+0000 7fd7bc000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7b4108f90 0x7fd7b419f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:49.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.365+0000 7fd7bc000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b410a180 0x7fd7b419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:49.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.365+0000 7fd7bc000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7b410ab40 0x7fd7b41a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:49.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.365+0000 7fd7bc000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b4114b20 con 0x7fd7b4108f90 2024-04-28T12:50:49.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.365+0000 7fd7bc000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b41149a0 con 0x7fd7b410a180 2024-04-28T12:50:49.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.365+0000 7fd7bc000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b4114ca0 con 0x7fd7b410ab40 2024-04-28T12:50:49.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.365+0000 7fd7b9600640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7b4108f90 0x7fd7b419f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.365+0000 7fd7b9600640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7b4108f90 0x7fd7b419f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35282/0 (socket says 172.21.15.84:35282) 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.366+0000 7fd7b9600640 1 -- 172.21.15.84:0/843638503 learned_addr learned my addr 172.21.15.84:0/843638503 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.366+0000 7fd7ba000640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7b410ab40 0x7fd7b41a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.366+0000 7fd7b8c00640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b410a180 0x7fd7b419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.366+0000 7fd7b9600640 1 -- 172.21.15.84:0/843638503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7b410ab40 msgr2=0x7fd7b41a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.366+0000 7fd7b9600640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7b410ab40 0x7fd7b41a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.366+0000 7fd7b9600640 1 -- 172.21.15.84:0/843638503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b410a180 msgr2=0x7fd7b419f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.366+0000 7fd7b9600640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b410a180 0x7fd7b419f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.366+0000 7fd7b9600640 1 -- 172.21.15.84:0/843638503 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b41a6dd0 con 0x7fd7b4108f90 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.366+0000 7fd7b9600640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7b4108f90 0x7fd7b419f200 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800d950 tx=0x7fd7a800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.366+0000 7fd7b2000640 1 -- 172.21.15.84:0/843638503 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8017070 con 0x7fd7b4108f90 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.366+0000 7fd7b2000640 1 -- 172.21.15.84:0/843638503 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7a8009c30 con 0x7fd7b4108f90 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.366+0000 7fd7bc000640 1 -- 172.21.15.84:0/843638503 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b41a70c0 con 0x7fd7b4108f90 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.367+0000 7fd7b2000640 1 -- 172.21.15.84:0/843638503 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a800f620 con 0x7fd7b4108f90 2024-04-28T12:50:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.367+0000 7fd7bc000640 1 -- 172.21.15.84:0/843638503 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b41a75e0 con 0x7fd7b4108f90 2024-04-28T12:50:49.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.368+0000 7fd7bc000640 1 -- 172.21.15.84:0/843638503 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7b4110380 con 0x7fd7b4108f90 2024-04-28T12:50:49.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.370+0000 7fd7b2000640 1 -- 172.21.15.84:0/843638503 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd7a801a070 con 0x7fd7b4108f90 2024-04-28T12:50:49.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.371+0000 7fd7b2000640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd79c0a3cc0 0x7fd79c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:49.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.371+0000 7fd7b2000640 1 -- 172.21.15.84:0/843638503 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fd7a80d0240 con 0x7fd7b4108f90 2024-04-28T12:50:49.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.371+0000 7fd7b8c00640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd79c0a3cc0 0x7fd79c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.372+0000 7fd7b8c00640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd79c0a3cc0 0x7fd79c0a6180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0031500 tx=0x7fd7a0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:49.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.376+0000 7fd7b2000640 1 -- 172.21.15.84:0/843638503 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd7a8009de0 con 0x7fd7b4108f90 2024-04-28T12:50:49.577 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.579+0000 7fd7bc000640 1 -- 172.21.15.84:0/843638503 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd7b4063530 con 0x7fd7b4108f90 2024-04-28T12:50:49.577 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.580+0000 7fd7b2000640 1 -- 172.21.15.84:0/843638503 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fd7a809c0f0 con 0x7fd7b4108f90 2024-04-28T12:50:49.578 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T12:50:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.583+0000 7fd7bc000640 1 -- 172.21.15.84:0/843638503 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd79c0a3cc0 msgr2=0x7fd79c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.583+0000 7fd7bc000640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd79c0a3cc0 0x7fd79c0a6180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0031500 tx=0x7fd7a0005a20 comp rx=0 tx=0).stop 2024-04-28T12:50:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.583+0000 7fd7bc000640 1 -- 172.21.15.84:0/843638503 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7b4108f90 msgr2=0x7fd7b419f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.583+0000 7fd7bc000640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7b4108f90 0x7fd7b419f200 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800d950 tx=0x7fd7a800de10 comp rx=0 tx=0).stop 2024-04-28T12:50:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.583+0000 7fd7bc000640 1 -- 172.21.15.84:0/843638503 shutdown_connections 2024-04-28T12:50:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.583+0000 7fd7bc000640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd79c0a3cc0 0x7fd79c0a6180 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.583+0000 7fd7bc000640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7b410ab40 0x7fd7b41a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.583+0000 7fd7bc000640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7b410a180 0x7fd7b419f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.583+0000 7fd7bc000640 1 --2- 172.21.15.84:0/843638503 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7b4108f90 0x7fd7b419f200 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.583+0000 7fd7bc000640 1 -- 172.21.15.84:0/843638503 >> 172.21.15.84:0/843638503 conn(0x7fd7b4077f30 msgr2=0x7fd7b410d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.584+0000 7fd7bc000640 1 -- 172.21.15.84:0/843638503 shutdown_connections 2024-04-28T12:50:49.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.584+0000 7fd7bc000640 1 -- 172.21.15.84:0/843638503 wait complete. 2024-04-28T12:50:49.598 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T12:50:49.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.805+0000 7f5ab8a00640 1 -- 172.21.15.84:0/2570278586 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5ab4103940 msgr2=0x7f5ab4103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:49.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.805+0000 7f5ab8a00640 1 --2- 172.21.15.84:0/2570278586 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5ab4103940 0x7f5ab4103dc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009900 tx=0x7f5a9c02a3f0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.805+0000 7f5ab8a00640 1 -- 172.21.15.84:0/2570278586 shutdown_connections 2024-04-28T12:50:49.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.805+0000 7f5ab8a00640 1 --2- 172.21.15.84:0/2570278586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab4104300 0x7f5ab410ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.805+0000 7f5ab8a00640 1 --2- 172.21.15.84:0/2570278586 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5ab4103940 0x7f5ab4103dc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.805+0000 7f5ab8a00640 1 --2- 172.21.15.84:0/2570278586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4102750 0x7f5ab4102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.805+0000 7f5ab8a00640 1 -- 172.21.15.84:0/2570278586 >> 172.21.15.84:0/2570278586 conn(0x7f5ab40fdcc0 msgr2=0x7f5ab4100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:49.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.806+0000 7f5ab8a00640 1 -- 172.21.15.84:0/2570278586 shutdown_connections 2024-04-28T12:50:49.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.806+0000 7f5ab8a00640 1 -- 172.21.15.84:0/2570278586 wait complete. 2024-04-28T12:50:49.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.807+0000 7f5ab8a00640 1 Processor -- start 2024-04-28T12:50:49.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.807+0000 7f5ab8a00640 1 -- start start 2024-04-28T12:50:49.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.807+0000 7f5ab8a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5ab4102750 0x7f5ab419ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:49.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.807+0000 7f5ab8a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab4103940 0x7f5ab419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:49.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.807+0000 7f5ab8a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4104300 0x7f5ab41a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:49.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.807+0000 7f5ab8a00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab410e2b0 con 0x7f5ab4102750 2024-04-28T12:50:49.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.807+0000 7f5ab8a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab410e130 con 0x7f5ab4103940 2024-04-28T12:50:49.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.807+0000 7f5ab8a00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab410e430 con 0x7f5ab4104300 2024-04-28T12:50:49.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.807+0000 7f5ab3e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4104300 0x7f5ab41a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:49.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.807+0000 7f5ab3e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4104300 0x7f5ab41a2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:54376/0 (socket says 172.21.15.84:54376) 2024-04-28T12:50:49.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.807+0000 7f5ab3e00640 1 -- 172.21.15.84:0/3900446991 learned_addr learned my addr 172.21.15.84:0/3900446991 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.808+0000 7f5aabe00640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab4103940 0x7f5ab419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.808+0000 7f5ab3400640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5ab4102750 0x7f5ab419ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.808+0000 7f5ab3e00640 1 -- 172.21.15.84:0/3900446991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab4103940 msgr2=0x7f5ab419b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.808+0000 7f5ab3e00640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab4103940 0x7f5ab419b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.808+0000 7f5ab3e00640 1 -- 172.21.15.84:0/3900446991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5ab4102750 msgr2=0x7f5ab419ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.808+0000 7f5ab3e00640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5ab4102750 0x7f5ab419ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.808+0000 7f5ab3e00640 1 -- 172.21.15.84:0/3900446991 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab41a27b0 con 0x7f5ab4104300 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.808+0000 7f5ab3e00640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4104300 0x7f5ab41a2140 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00eb10 tx=0x7f5aac00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.809+0000 7f5ab0c00640 1 -- 172.21.15.84:0/3900446991 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac004b20 con 0x7f5ab4104300 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.809+0000 7f5ab0c00640 1 -- 172.21.15.84:0/3900446991 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5aac004cc0 con 0x7f5ab4104300 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.809+0000 7f5ab0c00640 1 -- 172.21.15.84:0/3900446991 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac010670 con 0x7f5ab4104300 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.809+0000 7f5ab8a00640 1 -- 172.21.15.84:0/3900446991 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab41a2aa0 con 0x7f5ab4104300 2024-04-28T12:50:49.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.809+0000 7f5ab8a00640 1 -- 172.21.15.84:0/3900446991 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab41a2f70 con 0x7f5ab4104300 2024-04-28T12:50:49.808 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.810+0000 7f5ab8a00640 1 -- 172.21.15.84:0/3900446991 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab41a3200 con 0x7f5ab4104300 2024-04-28T12:50:49.808 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.812+0000 7f5ab0c00640 1 -- 172.21.15.84:0/3900446991 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5aac005020 con 0x7f5ab4104300 2024-04-28T12:50:49.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.813+0000 7f5ab0c00640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5a980a3cc0 0x7f5a980a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:49.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.813+0000 7f5ab0c00640 1 -- 172.21.15.84:0/3900446991 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f5aac0d0e60 con 0x7f5ab4104300 2024-04-28T12:50:49.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.813+0000 7f5ab3400640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5a980a3cc0 0x7f5a980a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:49.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.814+0000 7f5ab3400640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5a980a3cc0 0x7f5a980a6180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40095f0 tx=0x7f5aa4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:49.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:49.817+0000 7f5ab0c00640 1 -- 172.21.15.84:0/3900446991 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5aac09d340 con 0x7f5ab4104300 2024-04-28T12:50:49.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:49 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/924951447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:49.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:49 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/843638503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:49.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:49 smithi084 ceph-mon[28081]: pgmap v394: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:50.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.019+0000 7f5ab8a00640 1 -- 172.21.15.84:0/3900446991 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5ab41a33d0 con 0x7f5ab4104300 2024-04-28T12:50:50.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.020+0000 7f5ab0c00640 1 -- 172.21.15.84:0/3900446991 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f5aac09cc40 con 0x7f5ab4104300 2024-04-28T12:50:50.018 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 64 2024-04-28T12:50:50.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.023+0000 7f5ab8a00640 1 -- 172.21.15.84:0/3900446991 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5a980a3cc0 msgr2=0x7f5a980a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:50.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.023+0000 7f5ab8a00640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5a980a3cc0 0x7f5a980a6180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40095f0 tx=0x7f5aa4005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:50.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.023+0000 7f5ab8a00640 1 -- 172.21.15.84:0/3900446991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4104300 msgr2=0x7f5ab41a2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:50.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.023+0000 7f5ab8a00640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4104300 0x7f5ab41a2140 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00eb10 tx=0x7f5aac00c2d0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.023+0000 7f5ab8a00640 1 -- 172.21.15.84:0/3900446991 shutdown_connections 2024-04-28T12:50:50.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.023+0000 7f5ab8a00640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5a980a3cc0 0x7f5a980a6180 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.023+0000 7f5ab8a00640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4104300 0x7f5ab41a2140 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.023+0000 7f5ab8a00640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab4103940 0x7f5ab419b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.023+0000 7f5ab8a00640 1 --2- 172.21.15.84:0/3900446991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5ab4102750 0x7f5ab419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.023+0000 7f5ab8a00640 1 -- 172.21.15.84:0/3900446991 >> 172.21.15.84:0/3900446991 conn(0x7f5ab40fdcc0 msgr2=0x7f5ab4074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:50.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.023+0000 7f5ab8a00640 1 -- 172.21.15.84:0/3900446991 shutdown_connections 2024-04-28T12:50:50.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.023+0000 7f5ab8a00640 1 -- 172.21.15.84:0/3900446991 wait complete. 2024-04-28T12:50:50.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T12:50:50.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:49 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/924951447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:50.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:49 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/843638503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:50.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:49 smithi179 ceph-mon[63029]: pgmap v394: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:50.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.243+0000 7ffbbee00640 1 -- 172.21.15.84:0/4054336710 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbb8104470 msgr2=0x7ffbb810ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:50.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.243+0000 7ffbbee00640 1 --2- 172.21.15.84:0/4054336710 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbb8104470 0x7ffbb810ad00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009ca0 tx=0x7ffbb002a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.244+0000 7ffbbee00640 1 -- 172.21.15.84:0/4054336710 shutdown_connections 2024-04-28T12:50:50.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.244+0000 7ffbbee00640 1 --2- 172.21.15.84:0/4054336710 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbb8104470 0x7ffbb810ad00 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.244+0000 7ffbbee00640 1 --2- 172.21.15.84:0/4054336710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbb8103ab0 0x7ffbb8103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.244+0000 7ffbbee00640 1 --2- 172.21.15.84:0/4054336710 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb81028c0 0x7ffbb8102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.244+0000 7ffbbee00640 1 -- 172.21.15.84:0/4054336710 >> 172.21.15.84:0/4054336710 conn(0x7ffbb80fde30 msgr2=0x7ffbb8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:50.241 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.244+0000 7ffbbee00640 1 -- 172.21.15.84:0/4054336710 shutdown_connections 2024-04-28T12:50:50.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.244+0000 7ffbbee00640 1 -- 172.21.15.84:0/4054336710 wait complete. 2024-04-28T12:50:50.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbee00640 1 Processor -- start 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbee00640 1 -- start start 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbee00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbb81028c0 0x7ffbb819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbee00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbb8103ab0 0x7ffbb819b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbee00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb8104470 0x7ffbb81a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbee00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb810e3d0 con 0x7ffbb81028c0 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbee00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb810e250 con 0x7ffbb8103ab0 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbee00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb810e550 con 0x7ffbb8104470 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbda00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbb81028c0 0x7ffbb819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbda00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbb81028c0 0x7ffbb819ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35326/0 (socket says 172.21.15.84:35326) 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbda00640 1 -- 172.21.15.84:0/1626640996 learned_addr learned my addr 172.21.15.84:0/1626640996 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbe400640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb8104470 0x7ffbb81a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.245+0000 7ffbbd000640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbb8103ab0 0x7ffbb819b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.246+0000 7ffbbda00640 1 -- 172.21.15.84:0/1626640996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb8104470 msgr2=0x7ffbb81a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:50.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.246+0000 7ffbbda00640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb8104470 0x7ffbb81a2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.246+0000 7ffbbda00640 1 -- 172.21.15.84:0/1626640996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbb8103ab0 msgr2=0x7ffbb819b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:50.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.246+0000 7ffbbda00640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbb8103ab0 0x7ffbb819b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.246+0000 7ffbbda00640 1 -- 172.21.15.84:0/1626640996 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb81a28d0 con 0x7ffbb81028c0 2024-04-28T12:50:50.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.246+0000 7ffbbda00640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbb81028c0 0x7ffbb819ac90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ffba800e9c0 tx=0x7ffba800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:50.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.246+0000 7ffbb6000640 1 -- 172.21.15.84:0/1626640996 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba800cd80 con 0x7ffbb81028c0 2024-04-28T12:50:50.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.246+0000 7ffbb6000640 1 -- 172.21.15.84:0/1626640996 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffba80035f0 con 0x7ffbb81028c0 2024-04-28T12:50:50.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.246+0000 7ffbbee00640 1 -- 172.21.15.84:0/1626640996 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb81a2bc0 con 0x7ffbb81028c0 2024-04-28T12:50:50.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.246+0000 7ffbb6000640 1 -- 172.21.15.84:0/1626640996 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba8010690 con 0x7ffbb81028c0 2024-04-28T12:50:50.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.247+0000 7ffbbee00640 1 -- 172.21.15.84:0/1626640996 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb81a30c0 con 0x7ffbb81028c0 2024-04-28T12:50:50.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:49 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/924951447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:50:50.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:49 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/843638503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:50:50.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:49 smithi161 ceph-mon[62387]: pgmap v394: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:50.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.248+0000 7ffbbee00640 1 -- 172.21.15.84:0/1626640996 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbb81a32f0 con 0x7ffbb81028c0 2024-04-28T12:50:50.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.250+0000 7ffbb6000640 1 -- 172.21.15.84:0/1626640996 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ffba8004000 con 0x7ffbb81028c0 2024-04-28T12:50:50.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.251+0000 7ffbb6000640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffb9c0a3c40 0x7ffb9c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:50.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.251+0000 7ffbb6000640 1 -- 172.21.15.84:0/1626640996 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ffba80d0e60 con 0x7ffbb81028c0 2024-04-28T12:50:50.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.251+0000 7ffbbd000640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffb9c0a3c40 0x7ffb9c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:50.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.252+0000 7ffbbd000640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffb9c0a3c40 0x7ffb9c0a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ffbb819c190 tx=0x7ffba4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:50.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.256+0000 7ffbb6000640 1 -- 172.21.15.84:0/1626640996 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffba809d300 con 0x7ffbb81028c0 2024-04-28T12:50:50.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.455+0000 7ffbbee00640 1 -- 172.21.15.84:0/1626640996 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ffbb81a34c0 con 0x7ffbb81028c0 2024-04-28T12:50:50.453 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.456+0000 7ffbb6000640 1 -- 172.21.15.84:0/1626640996 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7ffba809cc00 con 0x7ffbb81028c0 2024-04-28T12:50:50.453 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T12:50:50.456 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.458+0000 7ffbbee00640 1 -- 172.21.15.84:0/1626640996 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffb9c0a3c40 msgr2=0x7ffb9c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:50.456 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.458+0000 7ffbbee00640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffb9c0a3c40 0x7ffb9c0a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ffbb819c190 tx=0x7ffba4005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:50.456 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.458+0000 7ffbbee00640 1 -- 172.21.15.84:0/1626640996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbb81028c0 msgr2=0x7ffbb819ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:50.456 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.458+0000 7ffbbee00640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbb81028c0 0x7ffbb819ac90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ffba800e9c0 tx=0x7ffba800ee80 comp rx=0 tx=0).stop 2024-04-28T12:50:50.456 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.458+0000 7ffbbee00640 1 -- 172.21.15.84:0/1626640996 shutdown_connections 2024-04-28T12:50:50.456 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.458+0000 7ffbbee00640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ffb9c0a3c40 0x7ffb9c0a6100 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.456 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.458+0000 7ffbbee00640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb8104470 0x7ffbb81a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.456 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.458+0000 7ffbbee00640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbb8103ab0 0x7ffbb819b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.457 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.458+0000 7ffbbee00640 1 --2- 172.21.15.84:0/1626640996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffbb81028c0 0x7ffbb819ac90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.457 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.458+0000 7ffbbee00640 1 -- 172.21.15.84:0/1626640996 >> 172.21.15.84:0/1626640996 conn(0x7ffbb80fde30 msgr2=0x7ffbb8074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:50.457 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.459+0000 7ffbbee00640 1 -- 172.21.15.84:0/1626640996 shutdown_connections 2024-04-28T12:50:50.457 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.459+0000 7ffbbee00640 1 -- 172.21.15.84:0/1626640996 wait complete. 2024-04-28T12:50:50.474 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:50:50.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.662+0000 7f6ee0200640 1 -- 172.21.15.84:0/2140691556 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ed8076970 msgr2=0x7f6ed8074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:50.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.662+0000 7f6ee0200640 1 --2- 172.21.15.84:0/2140691556 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ed8076970 0x7f6ed8074dc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8009930 tx=0x7f6ec802a420 comp rx=0 tx=0).stop 2024-04-28T12:50:50.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.662+0000 7f6ee0200640 1 -- 172.21.15.84:0/2140691556 shutdown_connections 2024-04-28T12:50:50.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.662+0000 7f6ee0200640 1 --2- 172.21.15.84:0/2140691556 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ed810ef50 0x7f6ed8111340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.662+0000 7f6ee0200640 1 --2- 172.21.15.84:0/2140691556 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ed8075390 0x7f6ed8075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.662+0000 7f6ee0200640 1 --2- 172.21.15.84:0/2140691556 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ed8076970 0x7f6ed8074dc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.662+0000 7f6ee0200640 1 -- 172.21.15.84:0/2140691556 >> 172.21.15.84:0/2140691556 conn(0x7f6ed80fdf10 msgr2=0x7f6ed8100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:50.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.662+0000 7f6ee0200640 1 -- 172.21.15.84:0/2140691556 shutdown_connections 2024-04-28T12:50:50.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.662+0000 7f6ee0200640 1 -- 172.21.15.84:0/2140691556 wait complete. 2024-04-28T12:50:50.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.663+0000 7f6ee0200640 1 Processor -- start 2024-04-28T12:50:50.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.663+0000 7f6ee0200640 1 -- start start 2024-04-28T12:50:50.662 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.663+0000 7f6ee0200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ed8075390 0x7f6ed819f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:50.662 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.663+0000 7f6ee0200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ed8076970 0x7f6ed819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:50.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.663+0000 7f6ee0200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ed810ef50 0x7f6ed81a67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:50.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.663+0000 7f6ee0200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed8114ad0 con 0x7f6ed810ef50 2024-04-28T12:50:50.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.663+0000 7f6ee0200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed8114950 con 0x7f6ed8075390 2024-04-28T12:50:50.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.663+0000 7f6ee0200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed8114c50 con 0x7f6ed8076970 2024-04-28T12:50:50.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.663+0000 7f6ede200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ed810ef50 0x7f6ed81a67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:50.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.663+0000 7f6ede200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ed810ef50 0x7f6ed81a67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35352/0 (socket says 172.21.15.84:35352) 2024-04-28T12:50:50.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.663+0000 7f6ede200640 1 -- 172.21.15.84:0/1811136672 learned_addr learned my addr 172.21.15.84:0/1811136672 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:50.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.663+0000 7f6edce00640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ed8076970 0x7f6ed819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:50.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6edd800640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ed8075390 0x7f6ed819f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:50.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6ede200640 1 -- 172.21.15.84:0/1811136672 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ed8076970 msgr2=0x7f6ed819f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:50.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6ede200640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ed8076970 0x7f6ed819f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.664 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6ede200640 1 -- 172.21.15.84:0/1811136672 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ed8075390 msgr2=0x7f6ed819f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:50.664 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6ede200640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ed8075390 0x7f6ed819f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.664 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6ede200640 1 -- 172.21.15.84:0/1811136672 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed81a6e30 con 0x7f6ed810ef50 2024-04-28T12:50:50.664 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6edd800640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ed8075390 0x7f6ed819f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:50.664 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6edce00640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ed8076970 0x7f6ed819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:50.664 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6ede200640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ed810ef50 0x7f6ed81a67c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000eb10 tx=0x7f6ed000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:50.664 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6ed6000640 1 -- 172.21.15.84:0/1811136672 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0004b20 con 0x7f6ed810ef50 2024-04-28T12:50:50.664 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6ed6000640 1 -- 172.21.15.84:0/1811136672 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ed0004cc0 con 0x7f6ed810ef50 2024-04-28T12:50:50.664 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6ed6000640 1 -- 172.21.15.84:0/1811136672 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0010670 con 0x7f6ed810ef50 2024-04-28T12:50:50.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.664+0000 7f6ee0200640 1 -- 172.21.15.84:0/1811136672 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed81a7120 con 0x7f6ed810ef50 2024-04-28T12:50:50.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.665+0000 7f6ee0200640 1 -- 172.21.15.84:0/1811136672 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed81a7620 con 0x7f6ed810ef50 2024-04-28T12:50:50.666 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.666+0000 7f6ee0200640 1 -- 172.21.15.84:0/1811136672 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed8105120 con 0x7f6ed810ef50 2024-04-28T12:50:50.666 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.668+0000 7f6ed6000640 1 -- 172.21.15.84:0/1811136672 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6ed0005020 con 0x7f6ed810ef50 2024-04-28T12:50:50.666 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.669+0000 7f6ed6000640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6ebc0a3cc0 0x7f6ebc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:50.666 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.669+0000 7f6ed6000640 1 -- 172.21.15.84:0/1811136672 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f6ed00d0f50 con 0x7f6ed810ef50 2024-04-28T12:50:50.666 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.669+0000 7f6edd800640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6ebc0a3cc0 0x7f6ebc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:50.667 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.670+0000 7f6edd800640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6ebc0a3cc0 0x7f6ebc0a6180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8006010 tx=0x7f6ec80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:50.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.673+0000 7f6ed6000640 1 -- 172.21.15.84:0/1811136672 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6ed009d3f0 con 0x7f6ed810ef50 2024-04-28T12:50:50.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.898+0000 7f6ee0200640 1 -- 172.21.15.84:0/1811136672 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ed802dad0 con 0x7f6ed810ef50 2024-04-28T12:50:50.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.900+0000 7f6ed6000640 1 -- 172.21.15.84:0/1811136672 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11884 (secure 0 0 0) 0x7f6ed009ccf0 con 0x7f6ed810ef50 2024-04-28T12:50:50.897 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:50:50.897 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6841/1196924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":1196924806},{"type":"v1","addr":"172.21.15.84:6841","nonce":1196924806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:34.334879+0000","modified":"2024-04-28T12:49:08.738419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24562,"mds_2":24545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6839/3812720642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":3812720642},{"type":"v1","addr":"172.21.15.161:6839","nonce":3812720642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24545,24562,24602]},"id":1}]} 2024-04-28T12:50:50.898 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 24 2024-04-28T12:50:50.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.902+0000 7f6ee0200640 1 -- 172.21.15.84:0/1811136672 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6ebc0a3cc0 msgr2=0x7f6ebc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:50.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.902+0000 7f6ee0200640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6ebc0a3cc0 0x7f6ebc0a6180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8006010 tx=0x7f6ec80314a0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.902+0000 7f6ee0200640 1 -- 172.21.15.84:0/1811136672 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ed810ef50 msgr2=0x7f6ed81a67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:50.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.902+0000 7f6ee0200640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ed810ef50 0x7f6ed81a67c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000eb10 tx=0x7f6ed000c2d0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.902+0000 7f6ee0200640 1 -- 172.21.15.84:0/1811136672 shutdown_connections 2024-04-28T12:50:50.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.902+0000 7f6ee0200640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f6ebc0a3cc0 0x7f6ebc0a6180 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.902+0000 7f6ee0200640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ed810ef50 0x7f6ed81a67c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.902+0000 7f6ee0200640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ed8076970 0x7f6ed819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.902+0000 7f6ee0200640 1 --2- 172.21.15.84:0/1811136672 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ed8075390 0x7f6ed819f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:50.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.902+0000 7f6ee0200640 1 -- 172.21.15.84:0/1811136672 >> 172.21.15.84:0/1811136672 conn(0x7f6ed80fdf10 msgr2=0x7f6ed81022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:50.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.902+0000 7f6ee0200640 1 -- 172.21.15.84:0/1811136672 shutdown_connections 2024-04-28T12:50:50.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:50.902+0000 7f6ee0200640 1 -- 172.21.15.84:0/1811136672 wait complete. 2024-04-28T12:50:50.919 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:50:51.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.102+0000 7f7ce6400640 1 -- 172.21.15.84:0/3776509601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ce0078530 msgr2=0x7f7ce0078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:51.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.102+0000 7f7ce6400640 1 --2- 172.21.15.84:0/3776509601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ce0078530 0x7f7ce0078990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009900 tx=0x7f7cd002a3f0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.102+0000 7f7ce6400640 1 -- 172.21.15.84:0/3776509601 shutdown_connections 2024-04-28T12:50:51.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.102+0000 7f7ce6400640 1 --2- 172.21.15.84:0/3776509601 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce0078ed0 0x7f7ce0079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.102+0000 7f7ce6400640 1 --2- 172.21.15.84:0/3776509601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ce0078530 0x7f7ce0078990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.102+0000 7f7ce6400640 1 --2- 172.21.15.84:0/3776509601 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ce00772f0 0x7f7ce0077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.100 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.102+0000 7f7ce6400640 1 -- 172.21.15.84:0/3776509601 >> 172.21.15.84:0/3776509601 conn(0x7f7ce00ffeb0 msgr2=0x7f7ce0102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:51.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.102+0000 7f7ce6400640 1 -- 172.21.15.84:0/3776509601 shutdown_connections 2024-04-28T12:50:51.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.102+0000 7f7ce6400640 1 -- 172.21.15.84:0/3776509601 wait complete. 2024-04-28T12:50:51.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.103+0000 7f7ce6400640 1 Processor -- start 2024-04-28T12:50:51.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.103+0000 7f7ce6400640 1 -- start start 2024-04-28T12:50:51.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce6400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce00772f0 0x7f7ce019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:51.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce6400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ce0078530 0x7f7ce019f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:51.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce6400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ce0078ed0 0x7f7ce01a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:51.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce6400640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce01148a0 con 0x7f7ce0078ed0 2024-04-28T12:50:51.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce6400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce0114720 con 0x7f7ce0078530 2024-04-28T12:50:51.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce6400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce0114a20 con 0x7f7ce00772f0 2024-04-28T12:50:51.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce5a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ce0078ed0 0x7f7ce01a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:51.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce5a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ce0078ed0 0x7f7ce01a6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35366/0 (socket says 172.21.15.84:35366) 2024-04-28T12:50:51.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce5a00640 1 -- 172.21.15.84:0/1174682417 learned_addr learned my addr 172.21.15.84:0/1174682417 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:51.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7cdfe00640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ce0078530 0x7f7ce019f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:51.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce5000640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce00772f0 0x7f7ce019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:51.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce5a00640 1 -- 172.21.15.84:0/1174682417 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce00772f0 msgr2=0x7f7ce019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:51.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce5a00640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce00772f0 0x7f7ce019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce5a00640 1 -- 172.21.15.84:0/1174682417 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ce0078530 msgr2=0x7f7ce019f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:51.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce5a00640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ce0078530 0x7f7ce019f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.104+0000 7f7ce5a00640 1 -- 172.21.15.84:0/1174682417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce01a6ba0 con 0x7f7ce0078ed0 2024-04-28T12:50:51.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.105+0000 7f7ce5a00640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ce0078ed0 0x7f7ce01a6530 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800eb10 tx=0x7f7cd800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:51.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.105+0000 7f7cdd600640 1 -- 172.21.15.84:0/1174682417 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8004b20 con 0x7f7ce0078ed0 2024-04-28T12:50:51.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.105+0000 7f7ce6400640 1 -- 172.21.15.84:0/1174682417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce01a6e90 con 0x7f7ce0078ed0 2024-04-28T12:50:51.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.105+0000 7f7cdd600640 1 -- 172.21.15.84:0/1174682417 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cd8004cc0 con 0x7f7ce0078ed0 2024-04-28T12:50:51.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.105+0000 7f7cdd600640 1 -- 172.21.15.84:0/1174682417 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8010690 con 0x7f7ce0078ed0 2024-04-28T12:50:51.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.105+0000 7f7ce6400640 1 -- 172.21.15.84:0/1174682417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce01a73a0 con 0x7f7ce0078ed0 2024-04-28T12:50:51.105 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.106+0000 7f7ce6400640 1 -- 172.21.15.84:0/1174682417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce01112d0 con 0x7f7ce0078ed0 2024-04-28T12:50:51.105 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.108+0000 7f7cdd600640 1 -- 172.21.15.84:0/1174682417 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7cd8005020 con 0x7f7ce0078ed0 2024-04-28T12:50:51.106 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.109+0000 7f7cdd600640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7cc80a3c40 0x7f7cc80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:51.106 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.109+0000 7f7cdd600640 1 -- 172.21.15.84:0/1174682417 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f7cd80d0e30 con 0x7f7ce0078ed0 2024-04-28T12:50:51.106 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.109+0000 7f7ce5000640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7cc80a3c40 0x7f7cc80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:51.107 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.110+0000 7f7ce5000640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7cc80a3c40 0x7f7cc80a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009bd0 tx=0x7f7cd4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:51.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.114+0000 7f7cdd600640 1 -- 172.21.15.84:0/1174682417 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7cd809d330 con 0x7f7ce0078ed0 2024-04-28T12:50:51.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:50 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3900446991' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:51.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:50 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1626640996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:50:51.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:50 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3900446991' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:51.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:50 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1626640996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:50:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:50 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3900446991' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:50:51.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:50 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1626640996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:50:51.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.327+0000 7f7ce6400640 1 -- 172.21.15.84:0/1174682417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ce0063530 con 0x7f7ce0078ed0 2024-04-28T12:50:51.328 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.331+0000 7f7cdd600640 1 -- 172.21.15.84:0/1174682417 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24634 (secure 0 0 0) 0x7f7cd809cc30 con 0x7f7ce0078ed0 2024-04-28T12:50:51.328 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:50:51.329 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":94,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:50:30.848698+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.8","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:50:51.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.333+0000 7f7ce6400640 1 -- 172.21.15.84:0/1174682417 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7cc80a3c40 msgr2=0x7f7cc80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:51.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.333+0000 7f7ce6400640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7cc80a3c40 0x7f7cc80a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009bd0 tx=0x7f7cd4005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:51.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.333+0000 7f7ce6400640 1 -- 172.21.15.84:0/1174682417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ce0078ed0 msgr2=0x7f7ce01a6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:51.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.333+0000 7f7ce6400640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ce0078ed0 0x7f7ce01a6530 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800eb10 tx=0x7f7cd800c2d0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.334+0000 7f7ce6400640 1 -- 172.21.15.84:0/1174682417 shutdown_connections 2024-04-28T12:50:51.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.334+0000 7f7ce6400640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7cc80a3c40 0x7f7cc80a6100 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.334+0000 7f7ce6400640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ce0078ed0 0x7f7ce01a6530 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.334+0000 7f7ce6400640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ce0078530 0x7f7ce019f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.334+0000 7f7ce6400640 1 --2- 172.21.15.84:0/1174682417 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7ce00772f0 0x7f7ce019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.334+0000 7f7ce6400640 1 -- 172.21.15.84:0/1174682417 >> 172.21.15.84:0/1174682417 conn(0x7f7ce00ffeb0 msgr2=0x7f7ce01019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:51.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.334+0000 7f7ce6400640 1 -- 172.21.15.84:0/1174682417 shutdown_connections 2024-04-28T12:50:51.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.334+0000 7f7ce6400640 1 -- 172.21.15.84:0/1174682417 wait complete. 2024-04-28T12:50:51.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:50:51.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.550+0000 7f4e09800640 1 -- 172.21.15.84:0/1565317499 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e04108f90 msgr2=0x7f4e041093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:51.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.550+0000 7f4e09800640 1 --2- 172.21.15.84:0/1565317499 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e04108f90 0x7f4e041093b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4df4009860 tx=0x7f4df402a340 comp rx=0 tx=0).stop 2024-04-28T12:50:51.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.550+0000 7f4e09800640 1 -- 172.21.15.84:0/1565317499 shutdown_connections 2024-04-28T12:50:51.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.550+0000 7f4e09800640 1 --2- 172.21.15.84:0/1565317499 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e0410ab40 0x7f4e041113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.550+0000 7f4e09800640 1 --2- 172.21.15.84:0/1565317499 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e0410a180 0x7f4e0410a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.550+0000 7f4e09800640 1 --2- 172.21.15.84:0/1565317499 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e04108f90 0x7f4e041093b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.550+0000 7f4e09800640 1 -- 172.21.15.84:0/1565317499 >> 172.21.15.84:0/1565317499 conn(0x7f4e04077f40 msgr2=0x7f4e0407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:51.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.551+0000 7f4e09800640 1 -- 172.21.15.84:0/1565317499 shutdown_connections 2024-04-28T12:50:51.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.551+0000 7f4e09800640 1 -- 172.21.15.84:0/1565317499 wait complete. 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4e09800640 1 Processor -- start 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4e09800640 1 -- start start 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4e09800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e04108f90 0x7f4e0419f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4e09800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e0410a180 0x7f4e0419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4e09800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e0410ab40 0x7f4e041a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4e09800640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f4e04114ad0 con 0x7f4e0410a180 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4e09800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4e04114950 con 0x7f4e04108f90 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4e09800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f4e04114c50 con 0x7f4e0410ab40 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4dfbe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e0410a180 0x7f4e0419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4dfbe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e0410a180 0x7f4e0419f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35380/0 (socket says 172.21.15.84:35380) 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4dfbe00640 1 -- 172.21.15.84:0/4037658698 learned_addr learned my addr 172.21.15.84:0/4037658698 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4e02a00640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e04108f90 0x7f4e0419f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:51.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.552+0000 7f4e03400640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e0410ab40 0x7f4e041a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:51.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.553+0000 7f4dfbe00640 1 -- 172.21.15.84:0/4037658698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e0410ab40 msgr2=0x7f4e041a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:51.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.553+0000 7f4dfbe00640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e0410ab40 0x7f4e041a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.553+0000 7f4dfbe00640 1 -- 172.21.15.84:0/4037658698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e04108f90 msgr2=0x7f4e0419f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:51.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.553+0000 7f4dfbe00640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e04108f90 0x7f4e0419f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.553+0000 7f4dfbe00640 1 -- 172.21.15.84:0/4037658698 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e041a6da0 con 0x7f4e0410a180 2024-04-28T12:50:51.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.553+0000 7f4e02a00640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e04108f90 0x7f4e0419f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:51.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.553+0000 7f4dfbe00640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e0410a180 0x7f4e0419f6c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00e9c0 tx=0x7f4dec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:51.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.553+0000 7f4dfb400640 1 -- 172.21.15.84:0/4037658698 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec00cde0 con 0x7f4e0410a180 2024-04-28T12:50:51.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.553+0000 7f4e09800640 1 -- 172.21.15.84:0/4037658698 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e041a7090 con 0x7f4e0410a180 2024-04-28T12:50:51.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.553+0000 7f4dfb400640 1 -- 172.21.15.84:0/4037658698 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4dec0035f0 con 0x7f4e0410a180 2024-04-28T12:50:51.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.553+0000 7f4dfb400640 1 -- 172.21.15.84:0/4037658698 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec010690 con 0x7f4e0410a180 2024-04-28T12:50:51.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.554+0000 7f4e09800640 1 -- 172.21.15.84:0/4037658698 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e041a75e0 con 0x7f4e0410a180 2024-04-28T12:50:51.553 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.555+0000 7f4e09800640 1 -- 172.21.15.84:0/4037658698 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e04110330 con 0x7f4e0410a180 2024-04-28T12:50:51.554 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.557+0000 7f4dfb400640 1 -- 172.21.15.84:0/4037658698 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4dec004000 con 0x7f4e0410a180 2024-04-28T12:50:51.555 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.558+0000 7f4dfb400640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4de80a3c40 0x7f4de80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:51.555 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.558+0000 7f4dfb400640 1 -- 172.21.15.84:0/4037658698 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f4dec0d0c60 con 0x7f4e0410a180 2024-04-28T12:50:51.555 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.558+0000 7f4e02a00640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4de80a3c40 0x7f4de80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:51.556 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.559+0000 7f4e02a00640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4de80a3c40 0x7f4de80a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4df4009830 tx=0x7f4df4030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:51.559 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.563+0000 7f4dfb400640 1 -- 172.21.15.84:0/4037658698 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4dec009510 con 0x7f4e0410a180 2024-04-28T12:50:51.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.812+0000 7f4e09800640 1 -- 172.21.15.84:0/4037658698 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e0402dad0 con 0x7f4e0410a180 2024-04-28T12:50:51.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.814+0000 7f4dfb400640 1 -- 172.21.15.84:0/4037658698 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11884 (secure 0 0 0) 0x7f4dec09d100 con 0x7f4e0410a180 2024-04-28T12:50:51.811 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:50:51.811 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6841/1196924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":1196924806},{"type":"v1","addr":"172.21.15.84:6841","nonce":1196924806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:34.334879+0000","modified":"2024-04-28T12:49:08.738419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24562,"mds_2":24545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6839/3812720642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":3812720642},{"type":"v1","addr":"172.21.15.161:6839","nonce":3812720642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24545,24562,24602]},"id":1}]} 2024-04-28T12:50:51.812 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 24 2024-04-28T12:50:51.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.816+0000 7f4e09800640 1 -- 172.21.15.84:0/4037658698 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4de80a3c40 msgr2=0x7f4de80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:51.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.816+0000 7f4e09800640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4de80a3c40 0x7f4de80a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4df4009830 tx=0x7f4df4030450 comp rx=0 tx=0).stop 2024-04-28T12:50:51.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.816+0000 7f4e09800640 1 -- 172.21.15.84:0/4037658698 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e0410a180 msgr2=0x7f4e0419f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:51.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.816+0000 7f4e09800640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e0410a180 0x7f4e0419f6c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00e9c0 tx=0x7f4dec00ee80 comp rx=0 tx=0).stop 2024-04-28T12:50:51.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.816+0000 7f4e09800640 1 -- 172.21.15.84:0/4037658698 shutdown_connections 2024-04-28T12:50:51.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.816+0000 7f4e09800640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f4de80a3c40 0x7f4de80a6100 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.816+0000 7f4e09800640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4e0410ab40 0x7f4e041a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.816+0000 7f4e09800640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e0410a180 0x7f4e0419f6c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.816+0000 7f4e09800640 1 --2- 172.21.15.84:0/4037658698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e04108f90 0x7f4e0419f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:51.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.817+0000 7f4e09800640 1 -- 172.21.15.84:0/4037658698 >> 172.21.15.84:0/4037658698 conn(0x7f4e04077f40 msgr2=0x7f4e0410d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:51.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.817+0000 7f4e09800640 1 -- 172.21.15.84:0/4037658698 shutdown_connections 2024-04-28T12:50:51.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:51.817+0000 7f4e09800640 1 -- 172.21.15.84:0/4037658698 wait complete. 2024-04-28T12:50:51.835 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-28T12:48:34.334879+0000', 'modified': '2024-04-28T12:49:08.738419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24602, 'mds_1': 24562, 'mds_2': 24545}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24509': {'gid': 24509, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/3348651842', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 3348651842}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 3348651842}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.179:6837/2376856149', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6836', 'nonce': 2376856149}, {'type': 'v1', 'addr': '172.21.15.179:6837', 'nonce': 2376856149}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24544': {'gid': 24544, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.179:6839/3879704230', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6838', 'nonce': 3879704230}, {'type': 'v1', 'addr': '172.21.15.179:6839', 'nonce': 3879704230}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24545': {'gid': 24545, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.161:6837/1346455857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 1346455857}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 1346455857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24562': {'gid': 24562, 'name': 'g', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.84:6839/3772865132', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6838', 'nonce': 3772865132}, {'type': 'v1', 'addr': '172.21.15.84:6839', 'nonce': 3772865132}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24602': {'gid': 24602, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6839/3812720642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 3812720642}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 3812720642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24602, 'qdb_cluster': [24545, 24562, 24602]} 2024-04-28T12:50:51.835 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-28T12:50:51.835 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-28T12:50:52.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.036+0000 7faa7b000640 1 -- 172.21.15.84:0/3957237726 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa74108d90 msgr2=0x7faa741091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.036+0000 7faa7b000640 1 --2- 172.21.15.84:0/3957237726 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa74108d90 0x7faa741091b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faa68009900 tx=0x7faa6802a400 comp rx=0 tx=0).stop 2024-04-28T12:50:52.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.036+0000 7faa7b000640 1 -- 172.21.15.84:0/3957237726 shutdown_connections 2024-04-28T12:50:52.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.036+0000 7faa7b000640 1 --2- 172.21.15.84:0/3957237726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa7410a940 0x7faa741111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.036+0000 7faa7b000640 1 --2- 172.21.15.84:0/3957237726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa74109f80 0x7faa7410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.036+0000 7faa7b000640 1 --2- 172.21.15.84:0/3957237726 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa74108d90 0x7faa741091b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.036+0000 7faa7b000640 1 -- 172.21.15.84:0/3957237726 >> 172.21.15.84:0/3957237726 conn(0x7faa74077d30 msgr2=0x7faa7407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.036+0000 7faa7b000640 1 -- 172.21.15.84:0/3957237726 shutdown_connections 2024-04-28T12:50:52.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.036+0000 7faa7b000640 1 -- 172.21.15.84:0/3957237726 wait complete. 2024-04-28T12:50:52.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.037+0000 7faa7b000640 1 Processor -- start 2024-04-28T12:50:52.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.038+0000 7faa7b000640 1 -- start start 2024-04-28T12:50:52.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.038+0000 7faa7b000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa74108d90 0x7faa7419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.038+0000 7faa7b000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa74109f80 0x7faa7419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.038+0000 7faa7b000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa7410a940 0x7faa741a6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.038+0000 7faa7b000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7faa74114910 con 0x7faa74109f80 2024-04-28T12:50:52.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.038+0000 7faa7b000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faa74114790 con 0x7faa74108d90 2024-04-28T12:50:52.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.038+0000 7faa7b000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7faa74114a90 con 0x7faa7410a940 2024-04-28T12:50:52.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.038+0000 7faa79200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa74109f80 0x7faa7419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.038+0000 7faa79200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa74109f80 0x7faa7419f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35394/0 (socket says 172.21.15.84:35394) 2024-04-28T12:50:52.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.038+0000 7faa7a600640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa7410a940 0x7faa741a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.038+0000 7faa79200640 1 -- 172.21.15.84:0/1000583377 learned_addr learned my addr 172.21.15.84:0/1000583377 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.038+0000 7faa79c00640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa74108d90 0x7faa7419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.039+0000 7faa7a600640 1 -- 172.21.15.84:0/1000583377 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa74108d90 msgr2=0x7faa7419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.039+0000 7faa7a600640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa74108d90 0x7faa7419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.039+0000 7faa7a600640 1 -- 172.21.15.84:0/1000583377 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa74109f80 msgr2=0x7faa7419f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.039+0000 7faa7a600640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa74109f80 0x7faa7419f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.039+0000 7faa7a600640 1 -- 172.21.15.84:0/1000583377 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa741a6bb0 con 0x7faa7410a940 2024-04-28T12:50:52.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.039+0000 7faa79c00640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa74108d90 0x7faa7419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:52.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.039+0000 7faa7a600640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa7410a940 0x7faa741a6540 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009fd0 tx=0x7faa6c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.039+0000 7faa72000640 1 -- 172.21.15.84:0/1000583377 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c00ce20 con 0x7faa7410a940 2024-04-28T12:50:52.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.039+0000 7faa72000640 1 -- 172.21.15.84:0/1000583377 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa6c005020 con 0x7faa7410a940 2024-04-28T12:50:52.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.039+0000 7faa72000640 1 -- 172.21.15.84:0/1000583377 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c010690 con 0x7faa7410a940 2024-04-28T12:50:52.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.039+0000 7faa7b000640 1 -- 172.21.15.84:0/1000583377 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa741a6ea0 con 0x7faa7410a940 2024-04-28T12:50:52.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.039+0000 7faa7b000640 1 -- 172.21.15.84:0/1000583377 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa741a7310 con 0x7faa7410a940 2024-04-28T12:50:52.048 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.043+0000 7faa7f44f740 1 Processor -- start 2024-04-28T12:50:52.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.044+0000 7faa72000640 1 -- 172.21.15.84:0/1000583377 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faa6c0054d0 con 0x7faa7410a940 2024-04-28T12:50:52.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.044+0000 7faa7f44f740 1 -- start start 2024-04-28T12:50:52.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.044+0000 7faa7f44f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.044+0000 7faa7f44f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c2cb0 0x564f000c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.044+0000 7faa7f44f740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c7e20 0x564f000ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.044+0000 7faa7f44f740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x564f00026600 con 0x564f000c7e20 2024-04-28T12:50:52.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.044+0000 7faa7f44f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x564efffc3100 con 0x564f000c22c0 2024-04-28T12:50:52.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.044+0000 7faa7f44f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564f000225c0 con 0x564f000c2cb0 2024-04-28T12:50:52.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.044+0000 7faa79200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c7e20 0x564f000ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa79c00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c2cb0 0x564f000c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa79200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c7e20 0x564f000ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35410/0 (socket says 172.21.15.84:35410) 2024-04-28T12:50:52.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa79200640 1 -- 172.21.15.84:0/688077072 learned_addr learned my addr 172.21.15.84:0/688077072 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa79200640 1 --2- 172.21.15.84:0/688077072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa79c00640 1 -- 172.21.15.84:0/688077072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 msgr2=0x564f000c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa79c00640 1 --2- 172.21.15.84:0/688077072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000c26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa79c00640 1 -- 172.21.15.84:0/688077072 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c7e20 msgr2=0x564f000ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa79c00640 1 --2- 172.21.15.84:0/688077072 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c7e20 0x564f000ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa79c00640 1 -- 172.21.15.84:0/688077072 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564effd382d0 con 0x564f000c2cb0 2024-04-28T12:50:52.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa72000640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7faa5c0a3cc0 0x7faa5c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa79200640 1 --2- 172.21.15.84:0/688077072 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c7e20 0x564f000ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:52.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa72000640 1 -- 172.21.15.84:0/1000583377 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7faa6c0d0ff0 con 0x7faa7410a940 2024-04-28T12:50:52.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.045+0000 7faa79c00640 1 --2- 172.21.15.84:0/688077072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c2cb0 0x564f000c77b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faa6802a920 tx=0x7faa68004140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=362ffce1b8477a36 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.046+0000 7faa79200640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7faa5c0a3cc0 0x7faa5c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.046+0000 7faa62a00640 1 -- 172.21.15.84:0/688077072 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa68035030 con 0x564f000c2cb0 2024-04-28T12:50:52.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.046+0000 7faa62a00640 1 -- 172.21.15.84:0/688077072 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa680054f0 con 0x564f000c2cb0 2024-04-28T12:50:52.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.046+0000 7faa62a00640 1 -- 172.21.15.84:0/688077072 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa680035f0 con 0x564f000c2cb0 2024-04-28T12:50:52.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.046+0000 7faa7f44f740 1 -- 172.21.15.84:0/688077072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c2cb0 msgr2=0x564f000c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.046+0000 7faa79200640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7faa5c0a3cc0 0x7faa5c0a6180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x564f000c3510 tx=0x7faa64004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.046+0000 7faa7f44f740 1 --2- 172.21.15.84:0/688077072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c2cb0 0x564f000c77b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faa6802a920 tx=0x7faa68004140 comp rx=0 tx=0).stop 2024-04-28T12:50:52.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.046+0000 7faa7f44f740 1 -- 172.21.15.84:0/688077072 shutdown_connections 2024-04-28T12:50:52.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.046+0000 7faa7f44f740 1 --2- 172.21.15.84:0/688077072 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c7e20 0x564f000ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.046+0000 7faa7f44f740 1 --2- 172.21.15.84:0/688077072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c2cb0 0x564f000c77b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.046+0000 7faa7f44f740 1 --2- 172.21.15.84:0/688077072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000c26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.047+0000 7faa7f44f740 1 -- 172.21.15.84:0/688077072 >> 172.21.15.84:0/688077072 conn(0x564f00056d10 msgr2=0x564f000bdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.047+0000 7faa7f44f740 1 -- 172.21.15.84:0/688077072 shutdown_connections 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.047+0000 7faa7f44f740 1 -- 172.21.15.84:0/688077072 wait complete. 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.047+0000 7faa7f44f740 1 -- start start 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.047+0000 7faa7f44f740 1 Processor -- start 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa7f44f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa7f44f740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 0x564f000ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa7f44f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000d3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa7f44f740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x564f000cad70 con 0x564f000c2cb0 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa7f44f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x564efffadf70 con 0x564f000c22c0 2024-04-28T12:50:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa7f44f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564f000caef0 con 0x564f000c7e20 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa7f44f740 10 client.0 _ll_register_callbacks cb 0x564f0001c480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa79200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 0x564f000ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa79c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa79200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 0x564f000ccc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35418/0 (socket says 172.21.15.84:35418) 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa79200640 1 -- 172.21.15.84:0/1992799885 learned_addr learned my addr 172.21.15.84:0/1992799885 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa7a600640 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000d3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa79200640 1 -- 172.21.15.84:0/1992799885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 msgr2=0x564f000d3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa79200640 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000d3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa79200640 1 -- 172.21.15.84:0/1992799885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 msgr2=0x564f000cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa79200640 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000cc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa79200640 1 -- 172.21.15.84:0/1992799885 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564effe026b0 con 0x564f000c2cb0 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.048+0000 7faa79c00640 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.049+0000 7faa79200640 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 0x564f000ccc30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faa6400efc0 tx=0x7faa64005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2861edce48ffd444 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.049+0000 7faa57400640 10 client.0 ms_handle_connect on v2:172.21.15.84:3300/0 2024-04-28T12:50:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.049+0000 7faa57400640 1 -- 172.21.15.84:0/1992799885 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6402d060 con 0x564f000c2cb0 2024-04-28T12:50:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.049+0000 7faa57400640 1 -- 172.21.15.84:0/1992799885 <== mon.0 v2:172.21.15.84:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7faa6401a070 con 0x564f000c2cb0 2024-04-28T12:50:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.049+0000 7faa7f44f740 1 -- 172.21.15.84:0/1992799885 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa680054f0 con 0x564f000c2cb0 2024-04-28T12:50:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.049+0000 7faa57400640 1 -- 172.21.15.84:0/1992799885 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faa64034050 con 0x564f000c2cb0 2024-04-28T12:50:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.049+0000 7faa7f44f740 10 client.15048 fetch_fsmap learned FSMap version 24 2024-04-28T12:50:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.049+0000 7faa7f44f740 1 -- 172.21.15.84:0/1992799885 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564f000d6cc0 con 0x564f000c2cb0 2024-04-28T12:50:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.050+0000 7faa57400640 1 -- 172.21.15.84:0/1992799885 <== mon.0 v2:172.21.15.84:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa680054f0 con 0x564f000c2cb0 2024-04-28T12:50:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.050+0000 7faa57400640 1 -- 172.21.15.84:0/1992799885 <== mon.0 v2:172.21.15.84:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7faa64027030 con 0x564f000c2cb0 2024-04-28T12:50:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.050+0000 7faa7f44f740 10 client.15048 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T12:50:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.050+0000 7faa7f44f740 10 client.15048 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-28T12:50:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.050+0000 7faa7f44f740 20 client.15048 populate_metadata read hostname 'smithi084' 2024-04-28T12:50:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.050+0000 7faa7f44f740 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x564f000d93a0 0x564f000da810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.050+0000 7faa7f44f740 4 client.15048 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:50:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.050+0000 7faa7f44f740 1 -- 172.21.15.84:0/1992799885 --> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f00034210 con 0x564f000d93a0 2024-04-28T12:50:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.050+0000 7faa79c00640 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x564f000d93a0 0x564f000da810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.051+0000 7faa79c00640 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x564f000d93a0 0x564f000da810 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c574a8aa460b1a2b server_cookie=bf2ebcdbae29fce4 in_seq=0 out_seq=0 2024-04-28T12:50:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.051+0000 7faa57400640 10 client.15048 ms_handle_connect on v2:172.21.15.161:6836/1346455857 2024-04-28T12:50:52.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.052+0000 7faa57400640 1 -- 172.21.15.84:0/1992799885 <== mds.2 v2:172.21.15.161:6836/1346455857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f00034210 con 0x564f000d93a0 2024-04-28T12:50:52.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.052+0000 7faa57400640 10 client.15048 handle_command_reply: tid=0 2024-04-28T12:50:52.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.052+0000 7faa7f44f740 1 client.15048 shutdown 2024-04-28T12:50:52.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.052+0000 7faa7f44f740 1 -- 172.21.15.84:0/1992799885 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 msgr2=0x564f000ccc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.052+0000 7faa7f44f740 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 0x564f000ccc30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faa6400efc0 tx=0x7faa64005f90 comp rx=0 tx=0).stop 2024-04-28T12:50:52.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.052+0000 7faa7f44f740 1 -- 172.21.15.84:0/1992799885 shutdown_connections 2024-04-28T12:50:52.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.052+0000 7faa7f44f740 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x564f000d93a0 0x564f000da810 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.052+0000 7faa7f44f740 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000d3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.052+0000 7faa7f44f740 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 0x564f000ccc30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.052+0000 7faa7f44f740 1 --2- 172.21.15.84:0/1992799885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.052+0000 7faa57400640 0 client.15048 ms_handle_reset on v2:172.21.15.161:6836/1346455857 2024-04-28T12:50:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.052+0000 7faa7f44f740 1 -- 172.21.15.84:0/1992799885 >> 172.21.15.84:0/1992799885 conn(0x564f00056d10 msgr2=0x564f000bcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.053+0000 7faa7f44f740 1 -- 172.21.15.84:0/1992799885 shutdown_connections 2024-04-28T12:50:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.053+0000 7faa7f44f740 1 -- 172.21.15.84:0/1992799885 wait complete. 2024-04-28T12:50:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.053+0000 7faa7f44f740 20 client.15048 trim_cache size 0 max 16384 2024-04-28T12:50:52.069 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.072+0000 7faa7f44f740 1 Processor -- start 2024-04-28T12:50:52.069 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.072+0000 7faa7f44f740 1 -- start start 2024-04-28T12:50:52.069 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.072+0000 7faa7f44f740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c22c0 0x564f000b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.072+0000 7faa7f44f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c2cb0 0x564f000d3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.072+0000 7faa7f44f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.072+0000 7faa7f44f740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x564f000d8b50 con 0x564f000c22c0 2024-04-28T12:50:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.072+0000 7faa7f44f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x564f000d9220 con 0x564f000c2cb0 2024-04-28T12:50:52.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.072+0000 7faa7f44f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564f000d98f0 con 0x564f000c7e20 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.073+0000 7faa79200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c2cb0 0x564f000d3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.073+0000 7faa79200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c2cb0 0x564f000d3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:52296/0 (socket says 172.21.15.84:52296) 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.073+0000 7faa79200640 1 -- 172.21.15.84:0/1282424949 learned_addr learned my addr 172.21.15.84:0/1282424949 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.073+0000 7faa7a600640 1 --2- 172.21.15.84:0/1282424949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.073+0000 7faa79c00640 1 --2- 172.21.15.84:0/1282424949 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c22c0 0x564f000b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.073+0000 7faa79200640 1 -- 172.21.15.84:0/1282424949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 msgr2=0x564f000cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.073+0000 7faa79200640 1 --2- 172.21.15.84:0/1282424949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.073+0000 7faa79200640 1 -- 172.21.15.84:0/1282424949 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c22c0 msgr2=0x564f000b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.073+0000 7faa79200640 1 --2- 172.21.15.84:0/1282424949 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c22c0 0x564f000b8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.073+0000 7faa79200640 1 -- 172.21.15.84:0/1282424949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f000ca670 con 0x564f000c2cb0 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.073+0000 7faa79200640 1 --2- 172.21.15.84:0/1282424949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c2cb0 0x564f000d3c30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faa6400ee50 tx=0x7faa64009c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a21b487ed1fe6389 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.073+0000 7faa62000640 1 -- 172.21.15.84:0/1282424949 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6402d070 con 0x564f000c2cb0 2024-04-28T12:50:52.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.074+0000 7faa62000640 1 -- 172.21.15.84:0/1282424949 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa640044e0 con 0x564f000c2cb0 2024-04-28T12:50:52.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.074+0000 7faa62000640 1 -- 172.21.15.84:0/1282424949 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa64003de0 con 0x564f000c2cb0 2024-04-28T12:50:52.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.074+0000 7faa7f44f740 1 -- 172.21.15.84:0/1282424949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c2cb0 msgr2=0x564f000d3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.074+0000 7faa7f44f740 1 --2- 172.21.15.84:0/1282424949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c2cb0 0x564f000d3c30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7faa6400ee50 tx=0x7faa64009c60 comp rx=0 tx=0).stop 2024-04-28T12:50:52.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.074+0000 7faa7f44f740 1 -- 172.21.15.84:0/1282424949 shutdown_connections 2024-04-28T12:50:52.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.074+0000 7faa7f44f740 1 --2- 172.21.15.84:0/1282424949 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.074+0000 7faa7f44f740 1 --2- 172.21.15.84:0/1282424949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c2cb0 0x564f000d3c30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.074+0000 7faa7f44f740 1 --2- 172.21.15.84:0/1282424949 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c22c0 0x564f000b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.074+0000 7faa7f44f740 1 -- 172.21.15.84:0/1282424949 >> 172.21.15.84:0/1282424949 conn(0x564f00056d10 msgr2=0x564f000c4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.074+0000 7faa7f44f740 1 -- 172.21.15.84:0/1282424949 shutdown_connections 2024-04-28T12:50:52.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.074+0000 7faa7f44f740 1 -- 172.21.15.84:0/1282424949 wait complete. 2024-04-28T12:50:52.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa7f44f740 1 -- start start 2024-04-28T12:50:52.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa7f44f740 1 Processor -- start 2024-04-28T12:50:52.077 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa7f44f740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.077 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa7f44f740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 0x564f000d0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.077 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa7f44f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000d3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.077 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa7f44f740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x564f000da690 con 0x564f000c2cb0 2024-04-28T12:50:52.077 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa7f44f740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x564f000d9fc0 con 0x564f000c22c0 2024-04-28T12:50:52.077 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa7f44f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x564f000dad60 con 0x564f000c7e20 2024-04-28T12:50:52.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa7f44f740 10 client.0 _ll_register_callbacks cb 0x7faa6400ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:52.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa79200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 0x564f000d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa7a600640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000d3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa79200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 0x564f000d0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35446/0 (socket says 172.21.15.84:35446) 2024-04-28T12:50:52.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa79200640 1 -- 172.21.15.84:0/2724906388 learned_addr learned my addr 172.21.15.84:0/2724906388 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.075+0000 7faa79c00640 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.076+0000 7faa7a600640 1 -- 172.21.15.84:0/2724906388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 msgr2=0x564f000cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.076+0000 7faa7a600640 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.076+0000 7faa7a600640 1 -- 172.21.15.84:0/2724906388 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 msgr2=0x564f000d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.076+0000 7faa7a600640 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 0x564f000d0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.076+0000 7faa7a600640 1 -- 172.21.15.84:0/2724906388 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f000d3610 con 0x564f000c7e20 2024-04-28T12:50:52.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.076+0000 7faa79200640 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 0x564f000d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.076+0000 7faa7a600640 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000d3020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faa6c0ce5e0 tx=0x7faa6c09ce70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f07d31139450f2f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.076+0000 7faa57400640 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.076+0000 7faa57400640 1 -- 172.21.15.84:0/2724906388 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c0ba020 con 0x564f000c7e20 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.076+0000 7faa57400640 1 -- 172.21.15.84:0/2724906388 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7faa6c0b6170 con 0x564f000c7e20 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.076+0000 7faa7f44f740 1 -- 172.21.15.84:0/2724906388 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa640044e0 con 0x564f000c7e20 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.077+0000 7faa57400640 1 -- 172.21.15.84:0/2724906388 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faa6c085050 con 0x564f000c7e20 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.077+0000 7faa7f44f740 10 client.25162 fetch_fsmap learned FSMap version 24 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.077+0000 7faa7f44f740 1 -- 172.21.15.84:0/2724906388 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564f0013fbc0 con 0x564f000c7e20 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.077+0000 7faa57400640 1 -- 172.21.15.84:0/2724906388 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa640044e0 con 0x564f000c7e20 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.077+0000 7faa57400640 1 -- 172.21.15.84:0/2724906388 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7faa6c0bd0b0 con 0x564f000c7e20 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.077+0000 7faa7f44f740 10 client.25162 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.077+0000 7faa7f44f740 10 client.25162 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.077+0000 7faa7f44f740 20 client.25162 populate_metadata read hostname 'smithi084' 2024-04-28T12:50:52.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.077+0000 7faa7f44f740 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x564f00144830 0x564f00140c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.077+0000 7faa7f44f740 4 client.25162 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.077+0000 7faa7f44f740 1 -- 172.21.15.84:0/2724906388 --> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] -- command(tid 0: {"prefix": "status"}) -- 0x564f0002e080 con 0x564f00144830 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.077+0000 7faa79c00640 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x564f00144830 0x564f00140c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa79c00640 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x564f00144830 0x564f00140c70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4b2f93eb147356a6 server_cookie=4a19fe83b8c22f1e in_seq=0 out_seq=0 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa57400640 10 client.25162 ms_handle_connect on v2:172.21.15.161:6836/1346455857 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa57400640 1 -- 172.21.15.84:0/2724906388 <== mds.2 v2:172.21.15.161:6836/1346455857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564f0002e080 con 0x564f00144830 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa57400640 10 client.25162 handle_command_reply: tid=0 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa7f44f740 1 client.25162 shutdown 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa7f44f740 1 -- 172.21.15.84:0/2724906388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 msgr2=0x564f000d3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa7f44f740 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000d3020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faa6c0ce5e0 tx=0x7faa6c09ce70 comp rx=0 tx=0).stop 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa7f44f740 1 -- 172.21.15.84:0/2724906388 shutdown_connections 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa7f44f740 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x564f00144830 0x564f00140c70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.080 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa7f44f740 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564f000c7e20 0x564f000d3020 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.081 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa7f44f740 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x564f000c2cb0 0x564f000d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.081 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa57400640 0 client.25162 ms_handle_reset on v2:172.21.15.161:6836/1346455857 2024-04-28T12:50:52.081 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa7f44f740 1 --2- 172.21.15.84:0/2724906388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564f000c22c0 0x564f000cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.081 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.078+0000 7faa7f44f740 1 -- 172.21.15.84:0/2724906388 >> 172.21.15.84:0/2724906388 conn(0x564f00056d10 msgr2=0x564f000cc300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.081 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7f44f740 1 -- 172.21.15.84:0/2724906388 shutdown_connections 2024-04-28T12:50:52.081 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7f44f740 1 -- 172.21.15.84:0/2724906388 wait complete. 2024-04-28T12:50:52.081 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7f44f740 20 client.25162 trim_cache size 0 max 16384 2024-04-28T12:50:52.081 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-28T12:50:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_fsid": "ddee604a-055b-11ef-bc93-c7b262605968", 2024-04-28T12:50:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "whoami": 2, 2024-04-28T12:50:52.081 INFO:teuthology.orchestra.run.smithi084.stdout: "id": 24545, 2024-04-28T12:50:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "want_state": "up:active", 2024-04-28T12:50:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "state": "up:active", 2024-04-28T12:50:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "fs_name": "cephfs", 2024-04-28T12:50:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "rank_uptime": 104.351375682, 2024-04-28T12:50:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "mdsmap_epoch": 24, 2024-04-28T12:50:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch": 94, 2024-04-28T12:50:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch_barrier": 90, 2024-04-28T12:50:52.082 INFO:teuthology.orchestra.run.smithi084.stdout: "uptime": 157.73692311400001 2024-04-28T12:50:52.082 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-28T12:50:52.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7b000640 1 -- 172.21.15.84:0/1000583377 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7faa5c0a3cc0 msgr2=0x7faa5c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7b000640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7faa5c0a3cc0 0x7faa5c0a6180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x564f000c3510 tx=0x7faa64004730 comp rx=0 tx=0).stop 2024-04-28T12:50:52.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7b000640 1 -- 172.21.15.84:0/1000583377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa7410a940 msgr2=0x7faa741a6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7b000640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa7410a940 0x7faa741a6540 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009fd0 tx=0x7faa6c00ef10 comp rx=0 tx=0).stop 2024-04-28T12:50:52.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7b000640 1 -- 172.21.15.84:0/1000583377 shutdown_connections 2024-04-28T12:50:52.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7b000640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7faa5c0a3cc0 0x7faa5c0a6180 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.083 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7b000640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa7410a940 0x7faa741a6540 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7b000640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa74109f80 0x7faa7419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7b000640 1 --2- 172.21.15.84:0/1000583377 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa74108d90 0x7faa7419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.079+0000 7faa7b000640 1 -- 172.21.15.84:0/1000583377 >> 172.21.15.84:0/1000583377 conn(0x7faa74077d30 msgr2=0x7faa7410d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.080+0000 7faa7b000640 1 -- 172.21.15.84:0/1000583377 shutdown_connections 2024-04-28T12:50:52.084 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.080+0000 7faa7b000640 1 -- 172.21.15.84:0/1000583377 wait complete. 2024-04-28T12:50:52.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:51 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1811136672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:52.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:51 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1174682417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:52.091 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:51 smithi084 ceph-mon[28081]: pgmap v395: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:52.093 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-28T12:50:52.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:51 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1811136672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:52.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:51 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1174682417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:52.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:51 smithi179 ceph-mon[63029]: pgmap v395: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:52.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:51 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1811136672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:52.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:51 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1174682417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:50:52.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:51 smithi161 ceph-mon[62387]: pgmap v395: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:52.286 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.288+0000 7fb0c5000640 1 -- 172.21.15.84:0/141944325 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c01027b0 msgr2=0x7fb0c0102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.286 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.288+0000 7fb0c5000640 1 --2- 172.21.15.84:0/141944325 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c01027b0 0x7fb0c0102bd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009ca0 tx=0x7fb0b802a8c0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.286 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.289+0000 7fb0c5000640 1 -- 172.21.15.84:0/141944325 shutdown_connections 2024-04-28T12:50:52.286 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.289+0000 7fb0c5000640 1 --2- 172.21.15.84:0/141944325 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c0104360 0x7fb0c010abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.289+0000 7fb0c5000640 1 --2- 172.21.15.84:0/141944325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0c01039a0 0x7fb0c0103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.289+0000 7fb0c5000640 1 --2- 172.21.15.84:0/141944325 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c01027b0 0x7fb0c0102bd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.289+0000 7fb0c5000640 1 -- 172.21.15.84:0/141944325 >> 172.21.15.84:0/141944325 conn(0x7fb0c00fdd40 msgr2=0x7fb0c0100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.289+0000 7fb0c5000640 1 -- 172.21.15.84:0/141944325 shutdown_connections 2024-04-28T12:50:52.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.289+0000 7fb0c5000640 1 -- 172.21.15.84:0/141944325 wait complete. 2024-04-28T12:50:52.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.290+0000 7fb0c5000640 1 Processor -- start 2024-04-28T12:50:52.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.290+0000 7fb0c5000640 1 -- start start 2024-04-28T12:50:52.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.290+0000 7fb0c5000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c01027b0 0x7fb0c00766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.290+0000 7fb0c5000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c01039a0 0x7fb0c0074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.290+0000 7fb0c5000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0c0104360 0x7fb0c0075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.290+0000 7fb0c5000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c010e330 con 0x7fb0c01039a0 2024-04-28T12:50:52.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.290+0000 7fb0c5000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c010e1b0 con 0x7fb0c0104360 2024-04-28T12:50:52.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.290+0000 7fb0c5000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c010e4b0 con 0x7fb0c01027b0 2024-04-28T12:50:52.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.290+0000 7fb0bea00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c01039a0 0x7fb0c0074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.290+0000 7fb0bf400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c01027b0 0x7fb0c00766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.291+0000 7fb0bf400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c01027b0 0x7fb0c00766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:54546/0 (socket says 172.21.15.84:54546) 2024-04-28T12:50:52.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.291+0000 7fb0bf400640 1 -- 172.21.15.84:0/1342034718 learned_addr learned my addr 172.21.15.84:0/1342034718 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.291+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0c0104360 0x7fb0c0075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.291+0000 7fb0bf400640 1 -- 172.21.15.84:0/1342034718 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0c0104360 msgr2=0x7fb0c0075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.291+0000 7fb0bf400640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0c0104360 0x7fb0c0075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.291+0000 7fb0bf400640 1 -- 172.21.15.84:0/1342034718 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c01039a0 msgr2=0x7fb0c0074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.291+0000 7fb0bf400640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c01039a0 0x7fb0c0074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.291+0000 7fb0bf400640 1 -- 172.21.15.84:0/1342034718 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c0075af0 con 0x7fb0c01027b0 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.291+0000 7fb0bea00640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c01039a0 0x7fb0c0074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.291+0000 7fb0bf400640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c01027b0 0x7fb0c00766e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8000f80 tx=0x7fb0b8003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.292+0000 7fb0b7e00640 1 -- 172.21.15.84:0/1342034718 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b802f380 con 0x7fb0c01027b0 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.292+0000 7fb0b7e00640 1 -- 172.21.15.84:0/1342034718 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0b80317c0 con 0x7fb0c01027b0 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.292+0000 7fb0b7e00640 1 -- 172.21.15.84:0/1342034718 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b802fc50 con 0x7fb0c01027b0 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.292+0000 7fb0c5000640 1 -- 172.21.15.84:0/1342034718 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c01a2aa0 con 0x7fb0c01027b0 2024-04-28T12:50:52.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.292+0000 7fb0c5000640 1 -- 172.21.15.84:0/1342034718 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c01a2fa0 con 0x7fb0c01027b0 2024-04-28T12:50:52.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.295+0000 7fb0c9462740 1 Processor -- start 2024-04-28T12:50:52.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.295+0000 7fb0c9462740 1 -- start start 2024-04-28T12:50:52.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.295+0000 7fb0b7e00640 1 -- 172.21.15.84:0/1342034718 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb0b8004040 con 0x7fb0c01027b0 2024-04-28T12:50:52.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.295+0000 7fb0c9462740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c02c0 0x561c213c06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.295+0000 7fb0c9462740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c0cb0 0x561c213c57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.295+0000 7fb0c9462740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c5e20 0x561c213c82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.295+0000 7fb0c9462740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x561c21324600 con 0x561c213c02c0 2024-04-28T12:50:52.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.295+0000 7fb0c9462740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561c212c1100 con 0x561c213c5e20 2024-04-28T12:50:52.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.295+0000 7fb0c9462740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561c213205c0 con 0x561c213c0cb0 2024-04-28T12:50:52.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.296+0000 7fb0bfe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c02c0 0x561c213c06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.296+0000 7fb0bfe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c02c0 0x561c213c06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35460/0 (socket says 172.21.15.84:35460) 2024-04-28T12:50:52.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.296+0000 7fb0bfe00640 1 -- 172.21.15.84:0/1934868936 learned_addr learned my addr 172.21.15.84:0/1934868936 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.296+0000 7fb0bea00640 1 --2- 172.21.15.84:0/1934868936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c5e20 0x561c213c82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.296+0000 7fb0bf400640 1 --2- 172.21.15.84:0/1934868936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c0cb0 0x561c213c57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.295 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.296+0000 7fb0bfe00640 1 -- 172.21.15.84:0/1934868936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c0cb0 msgr2=0x561c213c57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.295 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.296+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/1934868936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c0cb0 0x561c213c57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.295 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.296+0000 7fb0bfe00640 1 -- 172.21.15.84:0/1934868936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c5e20 msgr2=0x561c213c82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.295 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.296+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/1934868936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c5e20 0x561c213c82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.295 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.296+0000 7fb0bfe00640 1 -- 172.21.15.84:0/1934868936 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c210362d0 con 0x561c213c02c0 2024-04-28T12:50:52.295 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.296+0000 7fb0bf400640 1 --2- 172.21.15.84:0/1934868936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c0cb0 0x561c213c57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:52.295 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.297+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/1934868936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c02c0 0x561c213c06e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009c20 tx=0x7fb0b0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b3ee38c4ec6a87 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.298 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.297+0000 7fb0b4c00640 1 -- 172.21.15.84:0/1934868936 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b001e070 con 0x561c213c02c0 2024-04-28T12:50:52.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.297+0000 7fb0b4c00640 1 -- 172.21.15.84:0/1934868936 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0b0019ae0 con 0x561c213c02c0 2024-04-28T12:50:52.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.297+0000 7fb0b4c00640 1 -- 172.21.15.84:0/1934868936 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b000e960 con 0x561c213c02c0 2024-04-28T12:50:52.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.297+0000 7fb0b7e00640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0a40a3cc0 0x7fb0a40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.297+0000 7fb0b7e00640 1 -- 172.21.15.84:0/1342034718 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb0b80f1c70 con 0x7fb0c01027b0 2024-04-28T12:50:52.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.297+0000 7fb0bf400640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0a40a3cc0 0x7fb0a40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.298+0000 7fb0c9462740 1 -- 172.21.15.84:0/1934868936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c02c0 msgr2=0x561c213c06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.298+0000 7fb0c9462740 1 --2- 172.21.15.84:0/1934868936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c02c0 0x561c213c06e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009c20 tx=0x7fb0b0011810 comp rx=0 tx=0).stop 2024-04-28T12:50:52.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.298+0000 7fb0c9462740 1 -- 172.21.15.84:0/1934868936 shutdown_connections 2024-04-28T12:50:52.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.298+0000 7fb0c9462740 1 --2- 172.21.15.84:0/1934868936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c5e20 0x561c213c82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.298+0000 7fb0c9462740 1 --2- 172.21.15.84:0/1934868936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c0cb0 0x561c213c57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.298+0000 7fb0c9462740 1 --2- 172.21.15.84:0/1934868936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c02c0 0x561c213c06e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.298+0000 7fb0c9462740 1 -- 172.21.15.84:0/1934868936 >> 172.21.15.84:0/1934868936 conn(0x561c21354d10 msgr2=0x561c213bbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.298+0000 7fb0c9462740 1 -- 172.21.15.84:0/1934868936 shutdown_connections 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.298+0000 7fb0bf400640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0a40a3cc0 0x7fb0a40a6180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x561c213c10b0 tx=0x7fb0b802f780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.298+0000 7fb0c9462740 1 -- 172.21.15.84:0/1934868936 wait complete. 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.298+0000 7fb0c9462740 1 -- start start 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.298+0000 7fb0c9462740 1 Processor -- start 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.299+0000 7fb0c9462740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c02c0 0x561c213ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.299+0000 7fb0c9462740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c0cb0 0x561c213cac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.299+0000 7fb0c9462740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 0x561c213d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.299+0000 7fb0c9462740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x561c213c8dd0 con 0x561c213c5e20 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.299+0000 7fb0c9462740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561c212abf70 con 0x561c213c0cb0 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.299+0000 7fb0c9462740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561c213c8f50 con 0x561c213c02c0 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.299+0000 7fb0c9462740 10 client.0 _ll_register_callbacks cb 0x561c2131a480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.299+0000 7fb0bfe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 0x561c213d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.299+0000 7fb0bfe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 0x561c213d1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35466/0 (socket says 172.21.15.84:35466) 2024-04-28T12:50:52.300 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.299+0000 7fb0bfe00640 1 -- 172.21.15.84:0/1078474599 learned_addr learned my addr 172.21.15.84:0/1078474599 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.299+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c02c0 0x561c213ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.299+0000 7fb0bea00640 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c0cb0 0x561c213cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.300+0000 7fb0bfe00640 1 -- 172.21.15.84:0/1078474599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c02c0 msgr2=0x561c213ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.300+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c02c0 0x561c213ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.300+0000 7fb0bfe00640 1 -- 172.21.15.84:0/1078474599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c0cb0 msgr2=0x561c213cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.300+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c0cb0 0x561c213cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.300+0000 7fb0bfe00640 1 -- 172.21.15.84:0/1078474599 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c211006b0 con 0x561c213c5e20 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.300+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 0x561c213d1cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb0b001c040 tx=0x7fb0b000e3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea15420c8af8533 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.300+0000 7fb0ad600640 10 client.0 ms_handle_connect on v2:172.21.15.84:3300/0 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.300+0000 7fb0ad600640 1 -- 172.21.15.84:0/1078474599 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b001e030 con 0x561c213c5e20 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.300+0000 7fb0ad600640 1 -- 172.21.15.84:0/1078474599 <== mon.0 v2:172.21.15.84:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb0b0005480 con 0x561c213c5e20 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.300+0000 7fb0c9462740 1 -- 172.21.15.84:0/1078474599 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0b0019ae0 con 0x561c213c5e20 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.301+0000 7fb0ad600640 1 -- 172.21.15.84:0/1078474599 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb0b0035050 con 0x561c213c5e20 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.301+0000 7fb0c9462740 10 client.15066 fetch_fsmap learned FSMap version 24 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.301+0000 7fb0c9462740 1 -- 172.21.15.84:0/1078474599 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561c213d4d10 con 0x561c213c5e20 2024-04-28T12:50:52.301 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.301+0000 7fb0ad600640 1 -- 172.21.15.84:0/1078474599 <== mon.0 v2:172.21.15.84:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0b0019ae0 con 0x561c213c5e20 2024-04-28T12:50:52.302 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.301+0000 7fb0ad600640 1 -- 172.21.15.84:0/1078474599 <== mon.0 v2:172.21.15.84:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb0b000e660 con 0x561c213c5e20 2024-04-28T12:50:52.302 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.301+0000 7fb0c9462740 10 client.15066 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T12:50:52.302 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.301+0000 7fb0c9462740 10 client.15066 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-28T12:50:52.302 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.301+0000 7fb0c9462740 20 client.15066 populate_metadata read hostname 'smithi084' 2024-04-28T12:50:52.302 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.301+0000 7fb0c9462740 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x561c213d7180 0x561c213d85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.302 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.301+0000 7fb0c9462740 4 client.15066 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:50:52.302 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.301+0000 7fb0c9462740 1 -- 172.21.15.84:0/1078474599 --> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c21332210 con 0x561c213d7180 2024-04-28T12:50:52.302 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.302+0000 7fb0bea00640 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x561c213d7180 0x561c213d85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.303 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.302+0000 7fb0bea00640 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x561c213d7180 0x561c213d85f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c9f6de7bb538e1da server_cookie=ab83f37619bf931 in_seq=0 out_seq=0 2024-04-28T12:50:52.303 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.302+0000 7fb0ad600640 10 client.15066 ms_handle_connect on v2:172.21.15.84:6838/3772865132 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.303+0000 7fb0ad600640 1 -- 172.21.15.84:0/1078474599 <== mds.1 v2:172.21.15.84:6838/3772865132 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561c21332210 con 0x561c213d7180 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.303+0000 7fb0ad600640 10 client.15066 handle_command_reply: tid=0 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.303+0000 7fb0c9462740 1 client.15066 shutdown 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.303+0000 7fb0c9462740 1 -- 172.21.15.84:0/1078474599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 msgr2=0x561c213d1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.303+0000 7fb0c9462740 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 0x561c213d1cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb0b001c040 tx=0x7fb0b000e3b0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.304+0000 7fb0c9462740 1 -- 172.21.15.84:0/1078474599 shutdown_connections 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.304+0000 7fb0c9462740 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 0x561c213d1cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.304+0000 7fb0c9462740 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c0cb0 0x561c213cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.304+0000 7fb0c9462740 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x561c213d7180 0x561c213d85f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.304+0000 7fb0c9462740 1 --2- 172.21.15.84:0/1078474599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c02c0 0x561c213ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.304+0000 7fb0c9462740 1 -- 172.21.15.84:0/1078474599 >> 172.21.15.84:0/1078474599 conn(0x561c21354d10 msgr2=0x561c213baf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.304+0000 7fb0ad600640 0 client.15066 ms_handle_reset on v2:172.21.15.84:6838/3772865132 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.304+0000 7fb0c9462740 1 -- 172.21.15.84:0/1078474599 shutdown_connections 2024-04-28T12:50:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.304+0000 7fb0c9462740 1 -- 172.21.15.84:0/1078474599 wait complete. 2024-04-28T12:50:52.305 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.304+0000 7fb0c9462740 20 client.15066 trim_cache size 0 max 16384 2024-04-28T12:50:52.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.324+0000 7fb0c9462740 1 Processor -- start 2024-04-28T12:50:52.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.324+0000 7fb0c9462740 1 -- start start 2024-04-28T12:50:52.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.324+0000 7fb0c9462740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c02c0 0x561c213ca350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.324+0000 7fb0c9462740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c0cb0 0x561c213d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.324+0000 7fb0c9462740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c5e20 0x561c213ca890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.324+0000 7fb0c9462740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x561c213d6930 con 0x561c213c0cb0 2024-04-28T12:50:52.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.324+0000 7fb0c9462740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561c213d7000 con 0x561c213c02c0 2024-04-28T12:50:52.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.324+0000 7fb0c9462740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561c213d76d0 con 0x561c213c5e20 2024-04-28T12:50:52.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.324+0000 7fb0bea00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c0cb0 0x561c213d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.324+0000 7fb0bea00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c0cb0 0x561c213d1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35468/0 (socket says 172.21.15.84:35468) 2024-04-28T12:50:52.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.324+0000 7fb0bea00640 1 -- 172.21.15.84:0/2889103428 learned_addr learned my addr 172.21.15.84:0/2889103428 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.324+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/2889103428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c02c0 0x561c213ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.325+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/2889103428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c5e20 0x561c213ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.325+0000 7fb0bea00640 1 -- 172.21.15.84:0/2889103428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c5e20 msgr2=0x561c213ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.325+0000 7fb0bea00640 1 --2- 172.21.15.84:0/2889103428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c5e20 0x561c213ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.325+0000 7fb0bea00640 1 -- 172.21.15.84:0/2889103428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c02c0 msgr2=0x561c213ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.325+0000 7fb0bea00640 1 --2- 172.21.15.84:0/2889103428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c02c0 0x561c213ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.325+0000 7fb0bea00640 1 -- 172.21.15.84:0/2889103428 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c213c53b0 con 0x561c213c0cb0 2024-04-28T12:50:52.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.325+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/2889103428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c5e20 0x561c213ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.325+0000 7fb0bea00640 1 --2- 172.21.15.84:0/2889103428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c0cb0 0x561c213d1be0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8000c00 tx=0x7fb0a800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f39312cfab276a76 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.325+0000 7fb0afe00640 1 -- 172.21.15.84:0/2889103428 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8010b40 con 0x561c213c0cb0 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.325+0000 7fb0afe00640 1 -- 172.21.15.84:0/2889103428 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0a8024030 con 0x561c213c0cb0 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.325+0000 7fb0afe00640 1 -- 172.21.15.84:0/2889103428 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a801d6e0 con 0x561c213c0cb0 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.326+0000 7fb0c9462740 1 -- 172.21.15.84:0/2889103428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c0cb0 msgr2=0x561c213d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.326+0000 7fb0c9462740 1 --2- 172.21.15.84:0/2889103428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c0cb0 0x561c213d1be0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8000c00 tx=0x7fb0a800b410 comp rx=0 tx=0).stop 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.326+0000 7fb0c9462740 1 -- 172.21.15.84:0/2889103428 shutdown_connections 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.326+0000 7fb0c9462740 1 --2- 172.21.15.84:0/2889103428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c5e20 0x561c213ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.326+0000 7fb0c9462740 1 --2- 172.21.15.84:0/2889103428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c0cb0 0x561c213d1be0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.326+0000 7fb0c9462740 1 --2- 172.21.15.84:0/2889103428 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c02c0 0x561c213ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.326+0000 7fb0c9462740 1 -- 172.21.15.84:0/2889103428 >> 172.21.15.84:0/2889103428 conn(0x561c21354d10 msgr2=0x561c213d6170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.326+0000 7fb0c9462740 1 -- 172.21.15.84:0/2889103428 shutdown_connections 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.326+0000 7fb0c9462740 1 -- 172.21.15.84:0/2889103428 wait complete. 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.327+0000 7fb0c9462740 1 -- start start 2024-04-28T12:50:52.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.327+0000 7fb0c9462740 1 Processor -- start 2024-04-28T12:50:52.329 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.327+0000 7fb0c9462740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c02c0 0x561c213cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.329 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.327+0000 7fb0c9462740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c0cb0 0x561c213cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.329 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.327+0000 7fb0c9462740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 0x561c213d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.329 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.327+0000 7fb0c9462740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x561c213d8470 con 0x561c213c5e20 2024-04-28T12:50:52.329 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.327+0000 7fb0c9462740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x561c213d7da0 con 0x561c213c02c0 2024-04-28T12:50:52.329 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.327+0000 7fb0c9462740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x561c213d8b40 con 0x561c213c0cb0 2024-04-28T12:50:52.329 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.327+0000 7fb0c9462740 10 client.0 _ll_register_callbacks cb 0x7fb0b001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:52.329 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.328+0000 7fb0bfe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 0x561c213d0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.329 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.328+0000 7fb0bfe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 0x561c213d0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35478/0 (socket says 172.21.15.84:35478) 2024-04-28T12:50:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.328+0000 7fb0bfe00640 1 -- 172.21.15.84:0/2651358794 learned_addr learned my addr 172.21.15.84:0/2651358794 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.328+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c02c0 0x561c213cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.328+0000 7fb0bea00640 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c0cb0 0x561c213cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.328+0000 7fb0bfe00640 1 -- 172.21.15.84:0/2651358794 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c0cb0 msgr2=0x561c213cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.328+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c0cb0 0x561c213cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.328+0000 7fb0bfe00640 1 -- 172.21.15.84:0/2651358794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c02c0 msgr2=0x561c213cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.328+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c02c0 0x561c213cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.328+0000 7fb0bfe00640 1 -- 172.21.15.84:0/2651358794 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c213d1230 con 0x561c213c5e20 2024-04-28T12:50:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.328+0000 7fb0bfe00640 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 0x561c213d0c40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0001eb0 tx=0x7fb0b0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720869f144a6f186 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.328+0000 7fb0ad600640 10 client.0 ms_handle_connect on v2:172.21.15.84:3300/0 2024-04-28T12:50:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0ad600640 1 -- 172.21.15.84:0/2651358794 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b001e040 con 0x561c213c5e20 2024-04-28T12:50:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0ad600640 1 -- 172.21.15.84:0/2651358794 <== mon.0 v2:172.21.15.84:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb0b000ecf0 con 0x561c213c5e20 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0c9462740 1 -- 172.21.15.84:0/2651358794 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0a8024030 con 0x561c213c5e20 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0ad600640 1 -- 172.21.15.84:0/2651358794 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb0b0019850 con 0x561c213c5e20 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0c9462740 10 client.15072 fetch_fsmap learned FSMap version 24 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0c9462740 1 -- 172.21.15.84:0/2651358794 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561c21445d20 con 0x561c213c5e20 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0ad600640 1 -- 172.21.15.84:0/2651358794 <== mon.0 v2:172.21.15.84:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0a8024030 con 0x561c213c5e20 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0ad600640 1 -- 172.21.15.84:0/2651358794 <== mon.0 v2:172.21.15.84:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb0b0003680 con 0x561c213c5e20 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0c9462740 10 client.15072 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0c9462740 10 client.15072 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0c9462740 20 client.15072 populate_metadata read hostname 'smithi084' 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0c9462740 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x561c2144a8f0 0x561c21449600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0c9462740 4 client.15072 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.329+0000 7fb0c9462740 1 -- 172.21.15.84:0/2651358794 --> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] -- command(tid 0: {"prefix": "status"}) -- 0x561c2132c080 con 0x561c2144a8f0 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.330+0000 7fb0bea00640 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x561c2144a8f0 0x561c21449600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.330+0000 7fb0ad600640 10 client.15072 ms_handle_connect on v2:172.21.15.84:6838/3772865132 2024-04-28T12:50:52.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.330+0000 7fb0bea00640 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x561c2144a8f0 0x561c21449600 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3b434fbd1343c55e server_cookie=d9c8c7bf32442bbd in_seq=0 out_seq=0 2024-04-28T12:50:52.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0ad600640 1 -- 172.21.15.84:0/2651358794 <== mds.1 v2:172.21.15.84:6838/3772865132 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561c2132c080 con 0x561c2144a8f0 2024-04-28T12:50:52.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0ad600640 10 client.15072 handle_command_reply: tid=0 2024-04-28T12:50:52.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0c9462740 1 client.15072 shutdown 2024-04-28T12:50:52.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0c9462740 1 -- 172.21.15.84:0/2651358794 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 msgr2=0x561c213d0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0c9462740 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 0x561c213d0c40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0001eb0 tx=0x7fb0b0005850 comp rx=0 tx=0).stop 2024-04-28T12:50:52.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0c9462740 1 -- 172.21.15.84:0/2651358794 shutdown_connections 2024-04-28T12:50:52.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0c9462740 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x561c213c5e20 0x561c213d0c40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0c9462740 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x561c2144a8f0 0x561c21449600 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0c9462740 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561c213c0cb0 0x561c213cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0c9462740 1 --2- 172.21.15.84:0/2651358794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561c213c02c0 0x561c213cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0c9462740 1 -- 172.21.15.84:0/2651358794 >> 172.21.15.84:0/2651358794 conn(0x561c21354d10 msgr2=0x561c213ca350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0ad600640 0 client.15072 ms_handle_reset on v2:172.21.15.84:6838/3772865132 2024-04-28T12:50:52.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0c9462740 1 -- 172.21.15.84:0/2651358794 shutdown_connections 2024-04-28T12:50:52.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0c9462740 1 -- 172.21.15.84:0/2651358794 wait complete. 2024-04-28T12:50:52.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.331+0000 7fb0c9462740 20 client.15072 trim_cache size 0 max 16384 2024-04-28T12:50:52.333 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-28T12:50:52.333 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_fsid": "ddee604a-055b-11ef-bc93-c7b262605968", 2024-04-28T12:50:52.333 INFO:teuthology.orchestra.run.smithi084.stdout: "whoami": 1, 2024-04-28T12:50:52.333 INFO:teuthology.orchestra.run.smithi084.stdout: "id": 24562, 2024-04-28T12:50:52.334 INFO:teuthology.orchestra.run.smithi084.stdout: "want_state": "up:active", 2024-04-28T12:50:52.334 INFO:teuthology.orchestra.run.smithi084.stdout: "state": "up:active", 2024-04-28T12:50:52.334 INFO:teuthology.orchestra.run.smithi084.stdout: "fs_name": "cephfs", 2024-04-28T12:50:52.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rank_uptime": 105.616460188, 2024-04-28T12:50:52.334 INFO:teuthology.orchestra.run.smithi084.stdout: "mdsmap_epoch": 24, 2024-04-28T12:50:52.334 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch": 94, 2024-04-28T12:50:52.334 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch_barrier": 90, 2024-04-28T12:50:52.334 INFO:teuthology.orchestra.run.smithi084.stdout: "uptime": 150.76122713800001 2024-04-28T12:50:52.334 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-28T12:50:52.335 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.332+0000 7fb0c5000640 1 -- 172.21.15.84:0/1342034718 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0a40a3cc0 msgr2=0x7fb0a40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.335 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.332+0000 7fb0c5000640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0a40a3cc0 0x7fb0a40a6180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x561c213c10b0 tx=0x7fb0b802f780 comp rx=0 tx=0).stop 2024-04-28T12:50:52.335 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.332+0000 7fb0c5000640 1 -- 172.21.15.84:0/1342034718 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c01027b0 msgr2=0x7fb0c00766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.335 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.332+0000 7fb0c5000640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c01027b0 0x7fb0c00766e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8000f80 tx=0x7fb0b8003ba0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.335 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.332+0000 7fb0c5000640 1 -- 172.21.15.84:0/1342034718 shutdown_connections 2024-04-28T12:50:52.335 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.332+0000 7fb0c5000640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb0a40a3cc0 0x7fb0a40a6180 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.335 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.332+0000 7fb0c5000640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0c0104360 0x7fb0c0075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.335 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.332+0000 7fb0c5000640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c01039a0 0x7fb0c0074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.335 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.332+0000 7fb0c5000640 1 --2- 172.21.15.84:0/1342034718 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb0c01027b0 0x7fb0c00766e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.336 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.332+0000 7fb0c5000640 1 -- 172.21.15.84:0/1342034718 >> 172.21.15.84:0/1342034718 conn(0x7fb0c00fdd40 msgr2=0x7fb0c0106ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.336 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.332+0000 7fb0c5000640 1 -- 172.21.15.84:0/1342034718 shutdown_connections 2024-04-28T12:50:52.336 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.332+0000 7fb0c5000640 1 -- 172.21.15.84:0/1342034718 wait complete. 2024-04-28T12:50:52.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-28T12:50:52.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.559+0000 7ff637e00640 1 -- 172.21.15.84:0/715384444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff638076910 msgr2=0x7ff638074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.559+0000 7ff637e00640 1 --2- 172.21.15.84:0/715384444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff638076910 0x7ff638074d60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff628009860 tx=0x7ff62802a3d0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.559+0000 7ff637e00640 1 -- 172.21.15.84:0/715384444 shutdown_connections 2024-04-28T12:50:52.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.559+0000 7ff637e00640 1 --2- 172.21.15.84:0/715384444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff63810a5f0 0x7ff63810ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.559+0000 7ff637e00640 1 --2- 172.21.15.84:0/715384444 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff638075330 0x7ff638075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.559+0000 7ff637e00640 1 --2- 172.21.15.84:0/715384444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff638076910 0x7ff638074d60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.559+0000 7ff637e00640 1 -- 172.21.15.84:0/715384444 >> 172.21.15.84:0/715384444 conn(0x7ff6380fdcb0 msgr2=0x7ff638100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.559+0000 7ff637e00640 1 -- 172.21.15.84:0/715384444 shutdown_connections 2024-04-28T12:50:52.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.560+0000 7ff637e00640 1 -- 172.21.15.84:0/715384444 wait complete. 2024-04-28T12:50:52.558 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.561+0000 7ff637e00640 1 Processor -- start 2024-04-28T12:50:52.558 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.561+0000 7ff637e00640 1 -- start start 2024-04-28T12:50:52.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.561+0000 7ff637e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff638075330 0x7ff63819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.561+0000 7ff637e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff638076910 0x7ff63819b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.561+0000 7ff637e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff63810a5f0 0x7ff6381a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.561+0000 7ff637e00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff6381104d0 con 0x7ff63810a5f0 2024-04-28T12:50:52.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.561+0000 7ff637e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff638110350 con 0x7ff638075330 2024-04-28T12:50:52.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.561+0000 7ff637e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff638110650 con 0x7ff638076910 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.562+0000 7ff637400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff63810a5f0 0x7ff6381a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.562+0000 7ff62fe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff638076910 0x7ff63819b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.562+0000 7ff636a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff638075330 0x7ff63819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.562+0000 7ff62fe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff638076910 0x7ff63819b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:54582/0 (socket says 172.21.15.84:54582) 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.562+0000 7ff62fe00640 1 -- 172.21.15.84:0/4225936168 learned_addr learned my addr 172.21.15.84:0/4225936168 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.562+0000 7ff62fe00640 1 -- 172.21.15.84:0/4225936168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff638075330 msgr2=0x7ff63819abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.562+0000 7ff62fe00640 1 --2- 172.21.15.84:0/4225936168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff638075330 0x7ff63819abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.562+0000 7ff62fe00640 1 -- 172.21.15.84:0/4225936168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff63810a5f0 msgr2=0x7ff6381a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.562+0000 7ff62fe00640 1 --2- 172.21.15.84:0/4225936168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff63810a5f0 0x7ff6381a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.562+0000 7ff62fe00640 1 -- 172.21.15.84:0/4225936168 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6381a2780 con 0x7ff638076910 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.562+0000 7ff62fe00640 1 --2- 172.21.15.84:0/4225936168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff638076910 0x7ff63819b0f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff62000e9c0 tx=0x7ff62000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.563+0000 7ff62f400640 1 -- 172.21.15.84:0/4225936168 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62000ccc0 con 0x7ff638076910 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.563+0000 7ff62f400640 1 -- 172.21.15.84:0/4225936168 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6200054d0 con 0x7ff638076910 2024-04-28T12:50:52.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.563+0000 7ff62f400640 1 -- 172.21.15.84:0/4225936168 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff620010430 con 0x7ff638076910 2024-04-28T12:50:52.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.563+0000 7ff637e00640 1 -- 172.21.15.84:0/4225936168 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6381a2a70 con 0x7ff638076910 2024-04-28T12:50:52.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.563+0000 7ff637e00640 1 -- 172.21.15.84:0/4225936168 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6381a2f70 con 0x7ff638076910 2024-04-28T12:50:52.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.566+0000 7ff62f400640 1 -- 172.21.15.84:0/4225936168 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff620004f80 con 0x7ff638076910 2024-04-28T12:50:52.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.566+0000 7ff640a50740 1 Processor -- start 2024-04-28T12:50:52.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.566+0000 7ff640a50740 1 -- start start 2024-04-28T12:50:52.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.567+0000 7ff640a50740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a5174792c0 0x55a5174796e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.567+0000 7ff640a50740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a517479cb0 0x55a51747e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.567+0000 7ff640a50740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a5174812b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.567+0000 7ff640a50740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x55a5173dd600 con 0x55a5174792c0 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.567+0000 7ff640a50740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a51737a100 con 0x55a51747ee20 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.567+0000 7ff640a50740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a5173d95c0 con 0x55a517479cb0 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.567+0000 7ff637400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a5174792c0 0x55a5174796e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.567+0000 7ff636a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a517479cb0 0x55a51747e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.567+0000 7ff637400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a5174792c0 0x55a5174796e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35504/0 (socket says 172.21.15.84:35504) 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.567+0000 7ff636a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a517479cb0 0x55a51747e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:54598/0 (socket says 172.21.15.84:54598) 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.567+0000 7ff637400640 1 -- 172.21.15.84:0/2151304747 learned_addr learned my addr 172.21.15.84:0/2151304747 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.567+0000 7ff62fe00640 1 --2- 172.21.15.84:0/2151304747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a5174812b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff636a00640 1 -- 172.21.15.84:0/2151304747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 msgr2=0x55a5174812b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff636a00640 1 --2- 172.21.15.84:0/2151304747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a5174812b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff636a00640 1 -- 172.21.15.84:0/2151304747 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a5174792c0 msgr2=0x55a5174796e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff636a00640 1 --2- 172.21.15.84:0/2151304747 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a5174792c0 0x55a5174796e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff636a00640 1 -- 172.21.15.84:0/2151304747 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5170ef2d0 con 0x55a517479cb0 2024-04-28T12:50:52.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff637400640 1 --2- 172.21.15.84:0/2151304747 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a5174792c0 0x55a5174796e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:52.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff62f400640 1 --2- 172.21.15.84:0/4225936168 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff61c0a3cc0 0x7ff61c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff636a00640 1 --2- 172.21.15.84:0/2151304747 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a517479cb0 0x55a51747e7b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff628005e00 tx=0x7ff628030420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83f158b1a428cab1 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff62f400640 1 -- 172.21.15.84:0/4225936168 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff6200d0c40 con 0x7ff638076910 2024-04-28T12:50:52.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff637400640 1 --2- 172.21.15.84:0/4225936168 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff61c0a3cc0 0x7ff61c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff627e00640 1 -- 172.21.15.84:0/2151304747 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628030c80 con 0x55a517479cb0 2024-04-28T12:50:52.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff627e00640 1 -- 172.21.15.84:0/2151304747 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff628041070 con 0x55a517479cb0 2024-04-28T12:50:52.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.568+0000 7ff627e00640 1 -- 172.21.15.84:0/2151304747 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628004af0 con 0x55a517479cb0 2024-04-28T12:50:52.571 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.569+0000 7ff640a50740 1 -- 172.21.15.84:0/2151304747 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a517479cb0 msgr2=0x55a51747e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.571 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.569+0000 7ff637400640 1 --2- 172.21.15.84:0/4225936168 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff61c0a3cc0 0x7ff61c0a6180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55a51747a510 tx=0x7ff630009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.571 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.569+0000 7ff640a50740 1 --2- 172.21.15.84:0/2151304747 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a517479cb0 0x55a51747e7b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff628005e00 tx=0x7ff628030420 comp rx=0 tx=0).stop 2024-04-28T12:50:52.571 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.569+0000 7ff640a50740 1 -- 172.21.15.84:0/2151304747 shutdown_connections 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.569+0000 7ff640a50740 1 --2- 172.21.15.84:0/2151304747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a5174812b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.569+0000 7ff640a50740 1 --2- 172.21.15.84:0/2151304747 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a517479cb0 0x55a51747e7b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.569+0000 7ff640a50740 1 --2- 172.21.15.84:0/2151304747 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a5174792c0 0x55a5174796e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.569+0000 7ff640a50740 1 -- 172.21.15.84:0/2151304747 >> 172.21.15.84:0/2151304747 conn(0x55a51740dd10 msgr2=0x55a517474d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.569+0000 7ff640a50740 1 -- 172.21.15.84:0/2151304747 shutdown_connections 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.569+0000 7ff640a50740 1 -- 172.21.15.84:0/2151304747 wait complete. 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.570+0000 7ff640a50740 1 -- start start 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.570+0000 7ff640a50740 1 Processor -- start 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.570+0000 7ff640a50740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a5174792c0 0x55a5174836e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.570+0000 7ff640a50740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 0x55a517483c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.570+0000 7ff640a50740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a51748ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.570+0000 7ff640a50740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x55a517481d70 con 0x55a517479cb0 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.570+0000 7ff640a50740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a517364f70 con 0x55a51747ee20 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.570+0000 7ff640a50740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a517481ef0 con 0x55a5174792c0 2024-04-28T12:50:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff640a50740 10 client.0 _ll_register_callbacks cb 0x55a5173d3480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff62fe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 0x55a517483c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff62fe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 0x55a517483c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35506/0 (socket says 172.21.15.84:35506) 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff62fe00640 1 -- 172.21.15.84:0/2116192238 learned_addr learned my addr 172.21.15.84:0/2116192238 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff636a00640 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a5174792c0 0x55a5174836e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff637400640 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a51748ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff62fe00640 1 -- 172.21.15.84:0/2116192238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a5174792c0 msgr2=0x55a5174836e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff62fe00640 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a5174792c0 0x55a5174836e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff62fe00640 1 -- 172.21.15.84:0/2116192238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 msgr2=0x55a51748ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff62fe00640 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a51748ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff62fe00640 1 -- 172.21.15.84:0/2116192238 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5171b96b0 con 0x55a517479cb0 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff636a00640 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a5174792c0 0x55a5174836e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff62fe00640 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 0x55a517483c20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff6200a5180 tx=0x7ff620004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c65717f1e6872bb server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.571+0000 7ff624c00640 10 client.0 ms_handle_connect on v2:172.21.15.84:3300/0 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.572+0000 7ff624c00640 1 -- 172.21.15.84:0/2116192238 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62009c580 con 0x55a517479cb0 2024-04-28T12:50:52.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.572+0000 7ff624c00640 1 -- 172.21.15.84:0/2116192238 <== mon.0 v2:172.21.15.84:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff6200ce4c0 con 0x55a517479cb0 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.572+0000 7ff640a50740 1 -- 172.21.15.84:0/2116192238 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff628041070 con 0x55a517479cb0 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.572+0000 7ff624c00640 1 -- 172.21.15.84:0/2116192238 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff62000ce60 con 0x55a517479cb0 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.572+0000 7ff640a50740 10 client.15084 fetch_fsmap learned FSMap version 24 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.572+0000 7ff640a50740 1 -- 172.21.15.84:0/2116192238 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a51748dd10 con 0x55a517479cb0 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.572+0000 7ff624c00640 1 -- 172.21.15.84:0/2116192238 <== mon.0 v2:172.21.15.84:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff628041070 con 0x55a517479cb0 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.573+0000 7ff624c00640 1 -- 172.21.15.84:0/2116192238 <== mon.0 v2:172.21.15.84:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff620098790 con 0x55a517479cb0 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.573+0000 7ff640a50740 10 client.15084 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.573+0000 7ff640a50740 10 client.15084 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.573+0000 7ff640a50740 20 client.15084 populate_metadata read hostname 'smithi084' 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.573+0000 7ff640a50740 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55a5174903f0 0x55a517491860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.573+0000 7ff640a50740 4 client.15084 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.573+0000 7ff640a50740 1 -- 172.21.15.84:0/2116192238 --> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5173eb210 con 0x55a5174903f0 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.573+0000 7ff636a00640 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55a5174903f0 0x55a517491860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.574+0000 7ff636a00640 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55a5174903f0 0x55a517491860 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec432527276f76c1 server_cookie=3ea46c1fbedc64ce in_seq=0 out_seq=0 2024-04-28T12:50:52.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.574+0000 7ff624c00640 10 client.15084 ms_handle_connect on v2:172.21.15.161:6838/3812720642 2024-04-28T12:50:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff624c00640 1 -- 172.21.15.84:0/2116192238 <== mds.0 v2:172.21.15.161:6838/3812720642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a5173eb210 con 0x55a5174903f0 2024-04-28T12:50:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff624c00640 10 client.15084 handle_command_reply: tid=0 2024-04-28T12:50:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff640a50740 1 client.15084 shutdown 2024-04-28T12:50:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff640a50740 1 -- 172.21.15.84:0/2116192238 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 msgr2=0x55a517483c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff640a50740 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 0x55a517483c20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff6200a5180 tx=0x7ff620004a20 comp rx=0 tx=0).stop 2024-04-28T12:50:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff640a50740 1 -- 172.21.15.84:0/2116192238 shutdown_connections 2024-04-28T12:50:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff640a50740 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55a5174903f0 0x55a517491860 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff640a50740 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a51748ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff640a50740 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 0x55a517483c20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff640a50740 1 --2- 172.21.15.84:0/2116192238 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a5174792c0 0x55a5174836e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff640a50740 1 -- 172.21.15.84:0/2116192238 >> 172.21.15.84:0/2116192238 conn(0x55a51740dd10 msgr2=0x55a517473f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.577 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff624c00640 0 client.15084 ms_handle_reset on v2:172.21.15.161:6838/3812720642 2024-04-28T12:50:52.577 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff640a50740 1 -- 172.21.15.84:0/2116192238 shutdown_connections 2024-04-28T12:50:52.577 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff640a50740 1 -- 172.21.15.84:0/2116192238 wait complete. 2024-04-28T12:50:52.577 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.575+0000 7ff640a50740 20 client.15084 trim_cache size 0 max 16384 2024-04-28T12:50:52.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.595+0000 7ff640a50740 1 Processor -- start 2024-04-28T12:50:52.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.595+0000 7ff640a50740 1 -- start start 2024-04-28T12:50:52.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.595+0000 7ff640a50740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a5174792c0 0x55a51746fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.595+0000 7ff640a50740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 0x55a51748ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.595+0000 7ff640a50740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a51747ee20 0x55a5174832f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.595+0000 7ff640a50740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x55a51748fba0 con 0x55a517479cb0 2024-04-28T12:50:52.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.595+0000 7ff640a50740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a517490270 con 0x55a5174792c0 2024-04-28T12:50:52.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.595+0000 7ff640a50740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a517490940 con 0x55a51747ee20 2024-04-28T12:50:52.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.595+0000 7ff62fe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 0x55a51748ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.595+0000 7ff62fe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 0x55a51748ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35512/0 (socket says 172.21.15.84:35512) 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.595+0000 7ff62fe00640 1 -- 172.21.15.84:0/1092983061 learned_addr learned my addr 172.21.15.84:0/1092983061 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.596+0000 7ff637400640 1 --2- 172.21.15.84:0/1092983061 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a51747ee20 0x55a5174832f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.596+0000 7ff636a00640 1 --2- 172.21.15.84:0/1092983061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a5174792c0 0x55a51746fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.596+0000 7ff62fe00640 1 -- 172.21.15.84:0/1092983061 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a51747ee20 msgr2=0x55a5174832f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.596+0000 7ff62fe00640 1 --2- 172.21.15.84:0/1092983061 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a51747ee20 0x55a5174832f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.596+0000 7ff62fe00640 1 -- 172.21.15.84:0/1092983061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a5174792c0 msgr2=0x55a51746fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.596+0000 7ff62fe00640 1 --2- 172.21.15.84:0/1092983061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a5174792c0 0x55a51746fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.596+0000 7ff62fe00640 1 -- 172.21.15.84:0/1092983061 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a51747a540 con 0x55a517479cb0 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.596+0000 7ff636a00640 1 --2- 172.21.15.84:0/1092983061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a5174792c0 0x55a51746fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.596+0000 7ff62fe00640 1 --2- 172.21.15.84:0/1092983061 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 0x55a51748ac20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff620000c00 tx=0x7ff620098c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201bde8d8af161a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.596+0000 7ff627400640 1 -- 172.21.15.84:0/1092983061 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62009c3a0 con 0x55a517479cb0 2024-04-28T12:50:52.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.596+0000 7ff627400640 1 -- 172.21.15.84:0/1092983061 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6200bd0b0 con 0x55a517479cb0 2024-04-28T12:50:52.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.596+0000 7ff627400640 1 -- 172.21.15.84:0/1092983061 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6200a0df0 con 0x55a517479cb0 2024-04-28T12:50:52.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.597+0000 7ff640a50740 1 -- 172.21.15.84:0/1092983061 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 msgr2=0x55a51748ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.597+0000 7ff640a50740 1 --2- 172.21.15.84:0/1092983061 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 0x55a51748ac20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff620000c00 tx=0x7ff620098c40 comp rx=0 tx=0).stop 2024-04-28T12:50:52.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.597+0000 7ff640a50740 1 -- 172.21.15.84:0/1092983061 shutdown_connections 2024-04-28T12:50:52.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.597+0000 7ff640a50740 1 --2- 172.21.15.84:0/1092983061 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a51747ee20 0x55a5174832f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.597+0000 7ff640a50740 1 --2- 172.21.15.84:0/1092983061 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517479cb0 0x55a51748ac20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.597+0000 7ff640a50740 1 --2- 172.21.15.84:0/1092983061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a5174792c0 0x55a51746fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.597+0000 7ff640a50740 1 -- 172.21.15.84:0/1092983061 >> 172.21.15.84:0/1092983061 conn(0x55a51740dd10 msgr2=0x55a51747b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.597+0000 7ff640a50740 1 -- 172.21.15.84:0/1092983061 shutdown_connections 2024-04-28T12:50:52.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.597+0000 7ff640a50740 1 -- 172.21.15.84:0/1092983061 wait complete. 2024-04-28T12:50:52.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.598+0000 7ff640a50740 1 -- start start 2024-04-28T12:50:52.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.598+0000 7ff640a50740 1 Processor -- start 2024-04-28T12:50:52.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.598+0000 7ff640a50740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a5174792c0 0x55a5174865c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.598+0000 7ff640a50740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a517486b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.598+0000 7ff640a50740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517489b20 0x55a517489fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.598+0000 7ff640a50740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x55a5174916e0 con 0x55a517489b20 2024-04-28T12:50:52.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.598+0000 7ff640a50740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55a517491010 con 0x55a51747ee20 2024-04-28T12:50:52.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.598+0000 7ff640a50740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55a517491db0 con 0x55a5174792c0 2024-04-28T12:50:52.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.598+0000 7ff640a50740 10 client.0 _ll_register_callbacks cb 0x7ff6200d4480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff637400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517489b20 0x55a517489fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff637400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517489b20 0x55a517489fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35526/0 (socket says 172.21.15.84:35526) 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff637400640 1 -- 172.21.15.84:0/295814730 learned_addr learned my addr 172.21.15.84:0/295814730 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff62fe00640 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a517486b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff636a00640 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a5174792c0 0x55a5174865c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff637400640 1 -- 172.21.15.84:0/295814730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a5174792c0 msgr2=0x55a5174865c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff637400640 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a5174792c0 0x55a5174865c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff637400640 1 -- 172.21.15.84:0/295814730 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 msgr2=0x55a517486b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff637400640 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a517486b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff637400640 1 -- 172.21.15.84:0/295814730 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a51748a4e0 con 0x55a517489b20 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff62fe00640 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a517486b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff637400640 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517489b20 0x55a517489fa0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff630003f20 tx=0x7ff630004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e93c490d48ba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:52.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.599+0000 7ff624c00640 10 client.0 ms_handle_connect on v2:172.21.15.84:3300/0 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.600+0000 7ff624c00640 1 -- 172.21.15.84:0/295814730 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63002d060 con 0x55a517489b20 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.600+0000 7ff624c00640 1 -- 172.21.15.84:0/295814730 <== mon.0 v2:172.21.15.84:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff6300044c0 con 0x55a517489b20 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.600+0000 7ff640a50740 1 -- 172.21.15.84:0/295814730 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6200bd0b0 con 0x55a517489b20 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.600+0000 7ff624c00640 1 -- 172.21.15.84:0/295814730 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff630034050 con 0x55a517489b20 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.600+0000 7ff640a50740 10 client.15090 fetch_fsmap learned FSMap version 24 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.600+0000 7ff640a50740 1 -- 172.21.15.84:0/295814730 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a5174ff300 con 0x55a517489b20 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.600+0000 7ff624c00640 1 -- 172.21.15.84:0/295814730 <== mon.0 v2:172.21.15.84:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6200bd0b0 con 0x55a517489b20 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.601+0000 7ff624c00640 1 -- 172.21.15.84:0/295814730 <== mon.0 v2:172.21.15.84:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff630028420 con 0x55a517489b20 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.601+0000 7ff640a50740 10 client.15090 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.601+0000 7ff640a50740 10 client.15090 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.601+0000 7ff640a50740 20 client.15090 populate_metadata read hostname 'smithi084' 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.601+0000 7ff640a50740 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55a517503f50 0x55a517502c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:52.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.601+0000 7ff640a50740 4 client.15090 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.601+0000 7ff640a50740 1 -- 172.21.15.84:0/295814730 --> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] -- command(tid 0: {"prefix": "status"}) -- 0x55a5173e5080 con 0x55a517503f50 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.601+0000 7ff636a00640 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55a517503f50 0x55a517502c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff624c00640 10 client.15090 ms_handle_connect on v2:172.21.15.161:6838/3812720642 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff636a00640 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55a517503f50 0x55a517502c60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2e346d802f6cdb7 server_cookie=307b2df28c1f14a0 in_seq=0 out_seq=0 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff624c00640 1 -- 172.21.15.84:0/295814730 <== mds.0 v2:172.21.15.161:6838/3812720642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a5173e5080 con 0x55a517503f50 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff624c00640 10 client.15090 handle_command_reply: tid=0 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff640a50740 1 client.15090 shutdown 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff640a50740 1 -- 172.21.15.84:0/295814730 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517489b20 msgr2=0x55a517489fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff640a50740 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517489b20 0x55a517489fa0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff630003f20 tx=0x7ff630004a40 comp rx=0 tx=0).stop 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff640a50740 1 -- 172.21.15.84:0/295814730 shutdown_connections 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff640a50740 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55a517503f50 0x55a517502c60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff640a50740 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55a517489b20 0x55a517489fa0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff640a50740 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a51747ee20 0x55a517486b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.604 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff624c00640 0 client.15090 ms_handle_reset on v2:172.21.15.161:6838/3812720642 2024-04-28T12:50:52.604 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff640a50740 1 --2- 172.21.15.84:0/295814730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a5174792c0 0x55a5174865c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.604 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff640a50740 1 -- 172.21.15.84:0/295814730 >> 172.21.15.84:0/295814730 conn(0x55a51740dd10 msgr2=0x55a5174832f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.604 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff640a50740 1 -- 172.21.15.84:0/295814730 shutdown_connections 2024-04-28T12:50:52.604 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff640a50740 1 -- 172.21.15.84:0/295814730 wait complete. 2024-04-28T12:50:52.604 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.602+0000 7ff640a50740 20 client.15090 trim_cache size 0 max 16384 2024-04-28T12:50:52.604 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-28T12:50:52.604 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_fsid": "ddee604a-055b-11ef-bc93-c7b262605968", 2024-04-28T12:50:52.604 INFO:teuthology.orchestra.run.smithi084.stdout: "whoami": 0, 2024-04-28T12:50:52.604 INFO:teuthology.orchestra.run.smithi084.stdout: "id": 24602, 2024-04-28T12:50:52.604 INFO:teuthology.orchestra.run.smithi084.stdout: "want_state": "up:active", 2024-04-28T12:50:52.605 INFO:teuthology.orchestra.run.smithi084.stdout: "state": "up:active", 2024-04-28T12:50:52.605 INFO:teuthology.orchestra.run.smithi084.stdout: "fs_name": "cephfs", 2024-04-28T12:50:52.605 INFO:teuthology.orchestra.run.smithi084.stdout: "rank_uptime": 137.42478481699999, 2024-04-28T12:50:52.605 INFO:teuthology.orchestra.run.smithi084.stdout: "mdsmap_epoch": 24, 2024-04-28T12:50:52.605 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch": 94, 2024-04-28T12:50:52.605 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch_barrier": 90, 2024-04-28T12:50:52.605 INFO:teuthology.orchestra.run.smithi084.stdout: "uptime": 144.13146998400001 2024-04-28T12:50:52.605 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-28T12:50:52.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.603+0000 7ff637e00640 1 -- 172.21.15.84:0/4225936168 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff61c0a3cc0 msgr2=0x7ff61c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.603+0000 7ff637e00640 1 --2- 172.21.15.84:0/4225936168 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff61c0a3cc0 0x7ff61c0a6180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55a51747a510 tx=0x7ff630009500 comp rx=0 tx=0).stop 2024-04-28T12:50:52.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.603+0000 7ff637e00640 1 -- 172.21.15.84:0/4225936168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff638076910 msgr2=0x7ff63819b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:52.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.603+0000 7ff637e00640 1 --2- 172.21.15.84:0/4225936168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff638076910 0x7ff63819b0f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff62000e9c0 tx=0x7ff62000ee80 comp rx=0 tx=0).stop 2024-04-28T12:50:52.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.603+0000 7ff637e00640 1 -- 172.21.15.84:0/4225936168 shutdown_connections 2024-04-28T12:50:52.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.603+0000 7ff637e00640 1 --2- 172.21.15.84:0/4225936168 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff61c0a3cc0 0x7ff61c0a6180 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.603+0000 7ff637e00640 1 --2- 172.21.15.84:0/4225936168 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff63810a5f0 0x7ff6381a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.603+0000 7ff637e00640 1 --2- 172.21.15.84:0/4225936168 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff638076910 0x7ff63819b0f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.603+0000 7ff637e00640 1 --2- 172.21.15.84:0/4225936168 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff638075330 0x7ff63819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:52.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.603+0000 7ff637e00640 1 -- 172.21.15.84:0/4225936168 >> 172.21.15.84:0/4225936168 conn(0x7ff6380fdcb0 msgr2=0x7ff638106580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:52.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.603+0000 7ff637e00640 1 -- 172.21.15.84:0/4225936168 shutdown_connections 2024-04-28T12:50:52.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:52.603+0000 7ff637e00640 1 -- 172.21.15.84:0/4225936168 wait complete. 2024-04-28T12:50:52.619 INFO:tasks.cephadm:Setup complete, yielding 2024-04-28T12:50:52.619 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-28T12:50:52.629 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi084.front.sepia.ceph.com 2024-04-28T12:50:52.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-28T12:50:52.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:52 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4037658698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:53.074 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:53.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:52 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4037658698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:53.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:52 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4037658698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:50:53.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:53 smithi084 ceph-mon[28081]: pgmap v396: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:54.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:53 smithi179 ceph-mon[63029]: pgmap v396: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:54.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:53 smithi161 ceph-mon[62387]: pgmap v396: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:55.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.405+0000 7f42fbe00700 1 -- 172.21.15.84:0/1794838385 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42f4102370 msgr2=0x7f42f4102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:55.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.405+0000 7f42fbe00700 1 --2- 172.21.15.84:0/1794838385 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42f4102370 0x7f42f4102790 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f42e80099c0 tx=0x7f42e8009cd0 comp rx=0 tx=0).stop 2024-04-28T12:50:55.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.405+0000 7f42fbe00700 1 -- 172.21.15.84:0/1794838385 shutdown_connections 2024-04-28T12:50:55.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.405+0000 7f42fbe00700 1 --2- 172.21.15.84:0/1794838385 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42f4103f10 0x7f42f410a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:55.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.405+0000 7f42fbe00700 1 --2- 172.21.15.84:0/1794838385 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42f4102370 0x7f42f4102790 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:55.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.405+0000 7f42fbe00700 1 --2- 172.21.15.84:0/1794838385 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42f4103570 0x7f42f41039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:55.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.405+0000 7f42fbe00700 1 -- 172.21.15.84:0/1794838385 >> 172.21.15.84:0/1794838385 conn(0x7f42f40fd910 msgr2=0x7f42f40ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:55.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.405+0000 7f42fbe00700 1 -- 172.21.15.84:0/1794838385 shutdown_connections 2024-04-28T12:50:55.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.406+0000 7f42fbe00700 1 -- 172.21.15.84:0/1794838385 wait complete. 2024-04-28T12:50:55.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.407+0000 7f42fbe00700 1 Processor -- start 2024-04-28T12:50:55.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.407+0000 7f42fbe00700 1 -- start start 2024-04-28T12:50:55.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.407+0000 7f42fbe00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42f4102370 0x7f42f419b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:55.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.407+0000 7f42fbe00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42f4103570 0x7f42f419bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:55.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.407+0000 7f42fbe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42f4103f10 0x7f42f41a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:55.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.407+0000 7f42fbe00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f42f410de10 con 0x7f42f4103570 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.407+0000 7f42fbe00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f42f410dc90 con 0x7f42f4103f10 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.407+0000 7f42fbe00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f42f410df90 con 0x7f42f4102370 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.407+0000 7f42f9400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42f4102370 0x7f42f419b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.407+0000 7f42f8a00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42f4103570 0x7f42f419bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.407+0000 7f42f9400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42f4102370 0x7f42f419b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:54662/0 (socket says 172.21.15.84:54662) 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.407+0000 7f42f9400700 1 -- 172.21.15.84:0/1959009555 learned_addr learned my addr 172.21.15.84:0/1959009555 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.408+0000 7f42f9e00700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42f4103f10 0x7f42f41a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.408+0000 7f42f9400700 1 -- 172.21.15.84:0/1959009555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42f4103f10 msgr2=0x7f42f41a2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.408+0000 7f42f9400700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42f4103f10 0x7f42f41a2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.408+0000 7f42f9400700 1 -- 172.21.15.84:0/1959009555 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42f4103570 msgr2=0x7f42f419bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.408+0000 7f42f9400700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42f4103570 0x7f42f419bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.408+0000 7f42f9400700 1 -- 172.21.15.84:0/1959009555 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f41a3410 con 0x7f42f4102370 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.408+0000 7f42f9400700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42f4102370 0x7f42f419b7b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f42e8003b70 tx=0x7f42e8003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:55.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.409+0000 7f42f2000700 1 -- 172.21.15.84:0/1959009555 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8009e60 con 0x7f42f4102370 2024-04-28T12:50:55.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.409+0000 7f42fbe00700 1 -- 172.21.15.84:0/1959009555 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f41a36a0 con 0x7f42f4102370 2024-04-28T12:50:55.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.409+0000 7f42f2000700 1 -- 172.21.15.84:0/1959009555 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42e8005570 con 0x7f42f4102370 2024-04-28T12:50:55.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.409+0000 7f42f2000700 1 -- 172.21.15.84:0/1959009555 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e800fc40 con 0x7f42f4102370 2024-04-28T12:50:55.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.409+0000 7f42fbe00700 1 -- 172.21.15.84:0/1959009555 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f41a3bd0 con 0x7f42f4102370 2024-04-28T12:50:55.408 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.410+0000 7f42fbe00700 1 -- 172.21.15.84:0/1959009555 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f406a110 con 0x7f42f4102370 2024-04-28T12:50:55.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.412+0000 7f42f2000700 1 -- 172.21.15.84:0/1959009555 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f42e800f460 con 0x7f42f4102370 2024-04-28T12:50:55.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.414+0000 7f42f2000700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f42dc0a3d50 0x7f42dc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:55.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.414+0000 7f42f2000700 1 -- 172.21.15.84:0/1959009555 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f42e80d1750 con 0x7f42f4102370 2024-04-28T12:50:55.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.414+0000 7f42f8a00700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f42dc0a3d50 0x7f42dc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:55.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.415+0000 7f42f8a00700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f42dc0a3d50 0x7f42dc0a6210 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009c60 tx=0x7f42e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:55.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.416+0000 7f42f2000700 1 -- 172.21.15.84:0/1959009555 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f42e8013070 con 0x7f42f4102370 2024-04-28T12:50:55.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.665+0000 7f42fbe00700 1 -- 172.21.15.84:0/1959009555 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f42f40ffb20 con 0x7f42dc0a3d50 2024-04-28T12:50:55.665 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.668+0000 7f42f2000700 1 -- 172.21.15.84:0/1959009555 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f42f40ffb20 con 0x7f42dc0a3d50 2024-04-28T12:50:55.665 INFO:teuthology.orchestra.run.smithi084.stdout:Backend: cephadm 2024-04-28T12:50:55.666 INFO:teuthology.orchestra.run.smithi084.stdout:Available: Yes 2024-04-28T12:50:55.666 INFO:teuthology.orchestra.run.smithi084.stdout:Paused: No 2024-04-28T12:50:55.668 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.671+0000 7f42fbe00700 1 -- 172.21.15.84:0/1959009555 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f42dc0a3d50 msgr2=0x7f42dc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:55.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.671+0000 7f42fbe00700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f42dc0a3d50 0x7f42dc0a6210 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009c60 tx=0x7f42e0009260 comp rx=0 tx=0).stop 2024-04-28T12:50:55.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.671+0000 7f42fbe00700 1 -- 172.21.15.84:0/1959009555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42f4102370 msgr2=0x7f42f419b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:55.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.671+0000 7f42fbe00700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42f4102370 0x7f42f419b7b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f42e8003b70 tx=0x7f42e8003ba0 comp rx=0 tx=0).stop 2024-04-28T12:50:55.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.671+0000 7f42fbe00700 1 -- 172.21.15.84:0/1959009555 shutdown_connections 2024-04-28T12:50:55.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.671+0000 7f42fbe00700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f42dc0a3d50 0x7f42dc0a6210 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:55.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.671+0000 7f42fbe00700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42f4103f10 0x7f42f41a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:55.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.671+0000 7f42fbe00700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f42f4102370 0x7f42f419b7b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:55.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.671+0000 7f42fbe00700 1 --2- 172.21.15.84:0/1959009555 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f42f4103570 0x7f42f419bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:55.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.671+0000 7f42fbe00700 1 -- 172.21.15.84:0/1959009555 >> 172.21.15.84:0/1959009555 conn(0x7f42f40fd910 msgr2=0x7f42f40ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:55.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.671+0000 7f42fbe00700 1 -- 172.21.15.84:0/1959009555 shutdown_connections 2024-04-28T12:50:55.669 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:55.672+0000 7f42fbe00700 1 -- 172.21.15.84:0/1959009555 wait complete. 2024-04-28T12:50:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:55 smithi084 ceph-mon[28081]: pgmap v397: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:55.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:55 smithi084 ceph-mon[28081]: from='client.25204 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:55.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:55 smithi161 ceph-mon[62387]: pgmap v397: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:55.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:55 smithi161 ceph-mon[62387]: from='client.25204 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:56.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:55 smithi179 ceph-mon[63029]: pgmap v397: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:50:56.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:55 smithi179 ceph-mon[63029]: from='client.25204 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:56.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-28T12:50:56.981 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:50:57.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:57 smithi084 ceph-mon[28081]: pgmap v398: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:57.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:57 smithi161 ceph-mon[62387]: pgmap v398: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:58.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:57 smithi179 ceph-mon[63029]: pgmap v398: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:59.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.365+0000 7f5772c00700 1 -- 172.21.15.84:0/3233074303 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f576c103490 msgr2=0x7f576c1038b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:59.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.365+0000 7f5772c00700 1 --2- 172.21.15.84:0/3233074303 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f576c103490 0x7f576c1038b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f575c00b0d0 tx=0x7f575c00b3e0 comp rx=0 tx=0).stop 2024-04-28T12:50:59.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.365+0000 7f5772c00700 1 -- 172.21.15.84:0/3233074303 shutdown_connections 2024-04-28T12:50:59.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.365+0000 7f5772c00700 1 --2- 172.21.15.84:0/3233074303 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f576c103490 0x7f576c1038b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:59.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.365+0000 7f5772c00700 1 --2- 172.21.15.84:0/3233074303 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f576c105030 0x7f576c10b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:59.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.365+0000 7f5772c00700 1 --2- 172.21.15.84:0/3233074303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f576c104690 0x7f576c104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:59.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.365+0000 7f5772c00700 1 -- 172.21.15.84:0/3233074303 >> 172.21.15.84:0/3233074303 conn(0x7f576c0fe9f0 msgr2=0x7f576c100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:59.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.365+0000 7f5772c00700 1 -- 172.21.15.84:0/3233074303 shutdown_connections 2024-04-28T12:50:59.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.365+0000 7f5772c00700 1 -- 172.21.15.84:0/3233074303 wait complete. 2024-04-28T12:50:59.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.366+0000 7f5772c00700 1 Processor -- start 2024-04-28T12:50:59.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.366+0000 7f5772c00700 1 -- start start 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.366+0000 7f5772c00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f576c103490 0x7f576c19b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.366+0000 7f5772c00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f576c104690 0x7f576c19be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.366+0000 7f5772c00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f576c105030 0x7f576c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.366+0000 7f5772c00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f576c10ef90 con 0x7f576c103490 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.366+0000 7f5772c00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f576c10ee10 con 0x7f576c105030 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.366+0000 7f5772c00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f576c10f110 con 0x7f576c104690 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.367+0000 7f576be00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f576c103490 0x7f576c19b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.367+0000 7f5763400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f576c104690 0x7f576c19be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.367+0000 7f576be00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f576c103490 0x7f576c19b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41246/0 (socket says 172.21.15.84:41246) 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.367+0000 7f576be00700 1 -- 172.21.15.84:0/2907596018 learned_addr learned my addr 172.21.15.84:0/2907596018 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.367+0000 7f5770c00700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f576c105030 0x7f576c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.367+0000 7f576be00700 1 -- 172.21.15.84:0/2907596018 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f576c104690 msgr2=0x7f576c19be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.367+0000 7f576be00700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f576c104690 0x7f576c19be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:59.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.367+0000 7f576be00700 1 -- 172.21.15.84:0/2907596018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f576c105030 msgr2=0x7f576c1a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:59.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.367+0000 7f576be00700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f576c105030 0x7f576c1a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:59.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.367+0000 7f576be00700 1 -- 172.21.15.84:0/2907596018 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576c1a34c0 con 0x7f576c103490 2024-04-28T12:50:59.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.367+0000 7f5763400700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f576c104690 0x7f576c19be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:59.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.367+0000 7f5770c00700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f576c105030 0x7f576c1a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:50:59.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.368+0000 7f576be00700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f576c103490 0x7f576c19b900 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f575c004d00 tx=0x7f575c004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:59.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.368+0000 7f5769600700 1 -- 172.21.15.84:0/2907596018 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c01f030 con 0x7f576c103490 2024-04-28T12:50:59.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.368+0000 7f5772c00700 1 -- 172.21.15.84:0/2907596018 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f576c1a3750 con 0x7f576c103490 2024-04-28T12:50:59.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.368+0000 7f5769600700 1 -- 172.21.15.84:0/2907596018 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f575c005040 con 0x7f576c103490 2024-04-28T12:50:59.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.368+0000 7f5772c00700 1 -- 172.21.15.84:0/2907596018 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f576c1a3ba0 con 0x7f576c103490 2024-04-28T12:50:59.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.369+0000 7f5769600700 1 -- 172.21.15.84:0/2907596018 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c01b690 con 0x7f576c103490 2024-04-28T12:50:59.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.369+0000 7f5772c00700 1 -- 172.21.15.84:0/2907596018 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f576c06a110 con 0x7f576c103490 2024-04-28T12:50:59.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.371+0000 7f5769600700 1 -- 172.21.15.84:0/2907596018 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f575c022030 con 0x7f576c103490 2024-04-28T12:50:59.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.374+0000 7f5769600700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f57500a3cd0 0x7f57500a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:50:59.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.374+0000 7f5769600700 1 -- 172.21.15.84:0/2907596018 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f575c00f020 con 0x7f576c103490 2024-04-28T12:50:59.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.374+0000 7f5763400700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f57500a3cd0 0x7f57500a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:50:59.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.375+0000 7f5763400700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f57500a3cd0 0x7f57500a6190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5754011490 tx=0x7f5754009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:50:59.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.377+0000 7f5769600700 1 -- 172.21.15.84:0/2907596018 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f575c029020 con 0x7f576c103490 2024-04-28T12:50:59.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.615+0000 7f5772c00700 1 -- 172.21.15.84:0/2907596018 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f576c076d70 con 0x7f57500a3cd0 2024-04-28T12:50:59.629 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.631+0000 7f5769600700 1 -- 172.21.15.84:0/2907596018 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f576c076d70 con 0x7f57500a3cd0 2024-04-28T12:50:59.629 INFO:teuthology.orchestra.run.smithi084.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-28T12:50:59.629 INFO:teuthology.orchestra.run.smithi084.stdout:mds.a smithi084 running (2m) 34s ago 2m 12.3M - 19.0.0-3306-gbc612ffb 55dc54019ad8 d05f67529e91 2024-04-28T12:50:59.629 INFO:teuthology.orchestra.run.smithi084.stdout:mds.b smithi161 running (2m) 83s ago 2m 25.8M - 19.0.0-3306-gbc612ffb 55dc54019ad8 15ab11a559f9 2024-04-28T12:50:59.629 INFO:teuthology.orchestra.run.smithi084.stdout:mds.c smithi179 running (2m) 83s ago 2m 15.9M - 19.0.0-3306-gbc612ffb 55dc54019ad8 bd088b7c273c 2024-04-28T12:50:59.630 INFO:teuthology.orchestra.run.smithi084.stdout:mds.d smithi084 running (2m) 34s ago 2m 12.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 a1ee823846e8 2024-04-28T12:50:59.630 INFO:teuthology.orchestra.run.smithi084.stdout:mds.e smithi161 running (2m) 83s ago 2m 15.9M - 19.0.0-3306-gbc612ffb 55dc54019ad8 3492e64bd3bb 2024-04-28T12:50:59.630 INFO:teuthology.orchestra.run.smithi084.stdout:mds.f smithi179 running (2m) 83s ago 2m 14.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 0b9c806d90b4 2024-04-28T12:50:59.630 INFO:teuthology.orchestra.run.smithi084.stdout:mds.g smithi084 running (2m) 34s ago 2m 18.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 9db94e8bc7f0 2024-04-28T12:50:59.630 INFO:teuthology.orchestra.run.smithi084.stdout:mds.h smithi161 running (2m) 83s ago 2m 17.2M - 19.0.0-3306-gbc612ffb 55dc54019ad8 7c99db4ea817 2024-04-28T12:50:59.630 INFO:teuthology.orchestra.run.smithi084.stdout:mds.i smithi179 running (2m) 83s ago 2m 18.2M - 19.0.0-3306-gbc612ffb 55dc54019ad8 b20aa40ebd16 2024-04-28T12:50:59.630 INFO:teuthology.orchestra.run.smithi084.stdout:mds.j smithi084 running (3m) 34s ago 3m 12.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 80c5ddf58615 2024-04-28T12:50:59.630 INFO:teuthology.orchestra.run.smithi084.stdout:mds.k smithi161 running (3m) 83s ago 3m 16.3M - 19.0.0-3306-gbc612ffb 55dc54019ad8 5dbf90dedeae 2024-04-28T12:50:59.630 INFO:teuthology.orchestra.run.smithi084.stdout:mds.l smithi179 running (3m) 83s ago 3m 14.2M - 19.0.0-3306-gbc612ffb 55dc54019ad8 2c8bc769d5e5 2024-04-28T12:50:59.630 INFO:teuthology.orchestra.run.smithi084.stdout:mgr.x smithi084 *:9283,8765 running (11m) 34s ago 11m 515M - 19.0.0-3306-gbc612ffb 55dc54019ad8 9ede3f8332e6 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:mgr.y smithi161 *:8443,8765 running (9m) 83s ago 9m 454M - 19.0.0-3306-gbc612ffb 55dc54019ad8 5cbde99f5088 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:mgr.z smithi179 *:8443,8765 running (9m) 83s ago 9m 457M - 19.0.0-3306-gbc612ffb 55dc54019ad8 5fab2ce76a48 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:mon.a smithi084 running (11m) 34s ago 11m 49.6M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 d77f816c8b1c 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:mon.b smithi161 running (9m) 83s ago 9m 39.5M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 0411b65b5f43 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:mon.c smithi179 running (9m) 83s ago 9m 44.4M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 39e2a04435a1 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0 smithi084 running (8m) 34s ago 8m 44.9M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 404118554f76 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1 smithi161 running (8m) 83s ago 8m 56.2M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 68af435f8076 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2 smithi179 running (7m) 83s ago 7m 57.3M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 6ba367a23292 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 smithi084 running (7m) 34s ago 7m 53.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 51cb1d4caafb 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4 smithi161 running (6m) 83s ago 6m 55.6M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 f14b268e5d2c 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5 smithi179 running (6m) 83s ago 6m 56.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 7b2827a0ae7b 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6 smithi084 running (5m) 34s ago 5m 53.6M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 e74e5c1dc540 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7 smithi161 running (5m) 83s ago 5m 57.3M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 250a3bd2e622 2024-04-28T12:50:59.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8 smithi179 running (4m) 83s ago 4m 43.1M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 0ffb20db4201 2024-04-28T12:50:59.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9 smithi084 running (4m) 34s ago 4m 41.6M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 0ef79bcf44a0 2024-04-28T12:50:59.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10 smithi161 running (3m) 83s ago 3m 44.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 0029c0b09854 2024-04-28T12:50:59.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11 smithi179 running (3m) 83s ago 3m 43.4M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 20cddb7e2896 2024-04-28T12:50:59.632 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.634+0000 7f5772c00700 1 -- 172.21.15.84:0/2907596018 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f57500a3cd0 msgr2=0x7f57500a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:59.632 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.634+0000 7f5772c00700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f57500a3cd0 0x7f57500a6190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5754011490 tx=0x7f5754009260 comp rx=0 tx=0).stop 2024-04-28T12:50:59.633 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.634+0000 7f5772c00700 1 -- 172.21.15.84:0/2907596018 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f576c103490 msgr2=0x7f576c19b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:50:59.633 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.634+0000 7f5772c00700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f576c103490 0x7f576c19b900 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f575c004d00 tx=0x7f575c004d30 comp rx=0 tx=0).stop 2024-04-28T12:50:59.633 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.634+0000 7f5772c00700 1 -- 172.21.15.84:0/2907596018 shutdown_connections 2024-04-28T12:50:59.633 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.634+0000 7f5772c00700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f57500a3cd0 0x7f57500a6190 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:59.633 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.634+0000 7f5772c00700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f576c105030 0x7f576c1a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:59.633 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.634+0000 7f5772c00700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f576c103490 0x7f576c19b900 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:59.633 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.634+0000 7f5772c00700 1 --2- 172.21.15.84:0/2907596018 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f576c104690 0x7f576c19be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:50:59.633 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.634+0000 7f5772c00700 1 -- 172.21.15.84:0/2907596018 >> 172.21.15.84:0/2907596018 conn(0x7f576c0fe9f0 msgr2=0x7f576c075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:50:59.633 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.635+0000 7f5772c00700 1 -- 172.21.15.84:0/2907596018 shutdown_connections 2024-04-28T12:50:59.633 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:50:59.635+0000 7f5772c00700 1 -- 172.21.15.84:0/2907596018 wait complete. 2024-04-28T12:50:59.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:59 smithi084 ceph-mon[28081]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:59.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:50:59 smithi084 ceph-mon[28081]: pgmap v399: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:50:59.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:59 smithi161 ceph-mon[62387]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:50:59.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:50:59 smithi161 ceph-mon[62387]: pgmap v399: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:00.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:59 smithi179 ceph-mon[63029]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:00.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:50:59 smithi179 ceph-mon[63029]: pgmap v399: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:00.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-28T12:51:00.948 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:51:01.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:01 smithi084 ceph-mon[28081]: pgmap v400: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:01.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:01 smithi161 ceph-mon[62387]: pgmap v400: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:02.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:01 smithi179 ceph-mon[63029]: pgmap v400: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:03.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.316+0000 7f4062600700 1 -- 172.21.15.84:0/3971115198 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f405c077a40 msgr2=0x7f405c077ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:03.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.316+0000 7f4062600700 1 --2- 172.21.15.84:0/3971115198 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f405c077a40 0x7f405c077ec0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4048009990 tx=0x7f4048009ca0 comp rx=0 tx=0).stop 2024-04-28T12:51:03.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.316+0000 7f4062600700 1 -- 172.21.15.84:0/3971115198 shutdown_connections 2024-04-28T12:51:03.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.316+0000 7f4062600700 1 --2- 172.21.15.84:0/3971115198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c0794b0 0x7f405c0798d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:03.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.316+0000 7f4062600700 1 --2- 172.21.15.84:0/3971115198 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f405c078400 0x7f405c07a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:03.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.316+0000 7f4062600700 1 --2- 172.21.15.84:0/3971115198 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f405c077a40 0x7f405c077ec0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:03.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.316+0000 7f4062600700 1 -- 172.21.15.84:0/3971115198 >> 172.21.15.84:0/3971115198 conn(0x7f405c100c00 msgr2=0x7f405c103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:03.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.316+0000 7f4062600700 1 -- 172.21.15.84:0/3971115198 shutdown_connections 2024-04-28T12:51:03.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.316+0000 7f4062600700 1 -- 172.21.15.84:0/3971115198 wait complete. 2024-04-28T12:51:03.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.317+0000 7f4062600700 1 Processor -- start 2024-04-28T12:51:03.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.317+0000 7f4062600700 1 -- start start 2024-04-28T12:51:03.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.317+0000 7f4062600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c077a40 0x7f405c19fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.317+0000 7f4062600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f405c078400 0x7f405c1a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.317+0000 7f4062600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f405c0794b0 0x7f405c1a72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.317+0000 7f4062600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f405c1156a0 con 0x7f405c078400 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.317+0000 7f4062600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f405c115520 con 0x7f405c077a40 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.317+0000 7f4062600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f405c115820 con 0x7f405c0794b0 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.318+0000 7f405aa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f405c078400 0x7f405c1a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.318+0000 7f405aa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f405c078400 0x7f405c1a0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41268/0 (socket says 172.21.15.84:41268) 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.318+0000 7f405aa00700 1 -- 172.21.15.84:0/2905267146 learned_addr learned my addr 172.21.15.84:0/2905267146 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.318+0000 7f405b400700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c077a40 0x7f405c19fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.318+0000 7f405be00700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f405c0794b0 0x7f405c1a72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.318+0000 7f405aa00700 1 -- 172.21.15.84:0/2905267146 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f405c0794b0 msgr2=0x7f405c1a72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.318+0000 7f405aa00700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f405c0794b0 0x7f405c1a72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:03.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.318+0000 7f405aa00700 1 -- 172.21.15.84:0/2905267146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c077a40 msgr2=0x7f405c19fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:03.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.318+0000 7f405aa00700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c077a40 0x7f405c19fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:03.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.318+0000 7f405aa00700 1 -- 172.21.15.84:0/2905267146 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405c1a7970 con 0x7f405c078400 2024-04-28T12:51:03.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.318+0000 7f405b400700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c077a40 0x7f405c19fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:03.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.319+0000 7f405aa00700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f405c078400 0x7f405c1a0260 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f40480098e0 tx=0x7f40480057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:03.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.319+0000 7f4053e00700 1 -- 172.21.15.84:0/2905267146 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048027070 con 0x7f405c078400 2024-04-28T12:51:03.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.319+0000 7f4053e00700 1 -- 172.21.15.84:0/2905267146 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4048022470 con 0x7f405c078400 2024-04-28T12:51:03.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.319+0000 7f4062600700 1 -- 172.21.15.84:0/2905267146 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f405c1a7c00 con 0x7f405c078400 2024-04-28T12:51:03.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.319+0000 7f4053e00700 1 -- 172.21.15.84:0/2905267146 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048017650 con 0x7f405c078400 2024-04-28T12:51:03.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.319+0000 7f4062600700 1 -- 172.21.15.84:0/2905267146 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f405c1a80b0 con 0x7f405c078400 2024-04-28T12:51:03.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.320+0000 7f4062600700 1 -- 172.21.15.84:0/2905267146 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f405c06a110 con 0x7f405c078400 2024-04-28T12:51:03.319 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.322+0000 7f4053e00700 1 -- 172.21.15.84:0/2905267146 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f404802a070 con 0x7f405c078400 2024-04-28T12:51:03.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.324+0000 7f4053e00700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f40400a3d50 0x7f40400a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:03.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.324+0000 7f4053e00700 1 -- 172.21.15.84:0/2905267146 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f40480d17f0 con 0x7f405c078400 2024-04-28T12:51:03.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.324+0000 7f405b400700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f40400a3d50 0x7f40400a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:03.322 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.325+0000 7f405b400700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f40400a3d50 0x7f40400a6210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f405c1a1200 tx=0x7f404c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:03.325 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.328+0000 7f4053e00700 1 -- 172.21.15.84:0/2905267146 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f404805f060 con 0x7f405c078400 2024-04-28T12:51:03.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.563+0000 7f4062600700 1 -- 172.21.15.84:0/2905267146 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f405c02e870 con 0x7f40400a3d50 2024-04-28T12:51:03.571 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.573+0000 7f4053e00700 1 -- 172.21.15.84:0/2905267146 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f405c02e870 con 0x7f40400a3d50 2024-04-28T12:51:03.571 INFO:teuthology.orchestra.run.smithi084.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-28T12:51:03.571 INFO:teuthology.orchestra.run.smithi084.stdout:mds.all 12/12 87s ago 3m smithi084=a;smithi084=d;smithi084=g;smithi084=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l;count:12 2024-04-28T12:51:03.571 INFO:teuthology.orchestra.run.smithi084.stdout:mgr 3/3 87s ago 9m smithi084=x;smithi161=y;smithi179=z;count:3 2024-04-28T12:51:03.571 INFO:teuthology.orchestra.run.smithi084.stdout:mon 3/3 87s ago 9m smithi084:172.21.15.84=a;smithi161:172.21.15.161=b;smithi179:172.21.15.179=c;count:3 2024-04-28T12:51:03.571 INFO:teuthology.orchestra.run.smithi084.stdout:osd 12 87s ago - 2024-04-28T12:51:03.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.576+0000 7f4062600700 1 -- 172.21.15.84:0/2905267146 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f40400a3d50 msgr2=0x7f40400a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:03.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.576+0000 7f4062600700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f40400a3d50 0x7f40400a6210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f405c1a1200 tx=0x7f404c009260 comp rx=0 tx=0).stop 2024-04-28T12:51:03.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.576+0000 7f4062600700 1 -- 172.21.15.84:0/2905267146 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f405c078400 msgr2=0x7f405c1a0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:03.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.576+0000 7f4062600700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f405c078400 0x7f405c1a0260 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f40480098e0 tx=0x7f40480057c0 comp rx=0 tx=0).stop 2024-04-28T12:51:03.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.576+0000 7f4062600700 1 -- 172.21.15.84:0/2905267146 shutdown_connections 2024-04-28T12:51:03.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.577+0000 7f4062600700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f40400a3d50 0x7f40400a6210 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:03.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.577+0000 7f4062600700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c077a40 0x7f405c19fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:03.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.577+0000 7f4062600700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f405c0794b0 0x7f405c1a72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:03.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.577+0000 7f4062600700 1 --2- 172.21.15.84:0/2905267146 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f405c078400 0x7f405c1a0260 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:03.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.577+0000 7f4062600700 1 -- 172.21.15.84:0/2905267146 >> 172.21.15.84:0/2905267146 conn(0x7f405c100c00 msgr2=0x7f405c103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:03.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.577+0000 7f4062600700 1 -- 172.21.15.84:0/2905267146 shutdown_connections 2024-04-28T12:51:03.575 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:03.577+0000 7f4062600700 1 -- 172.21.15.84:0/2905267146 wait complete. 2024-04-28T12:51:03.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:03 smithi084 ceph-mon[28081]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:03.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:03 smithi084 ceph-mon[28081]: pgmap v401: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:03 smithi161 ceph-mon[62387]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:03 smithi161 ceph-mon[62387]: pgmap v401: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:04.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:03 smithi179 ceph-mon[63029]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:04.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:03 smithi179 ceph-mon[63029]: pgmap v401: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:04.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-28T12:51:04.904 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:51:05.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:05 smithi161 ceph-mon[62387]: pgmap v402: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:05.996 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:05 smithi084 ceph-mon[28081]: pgmap v402: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:06.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:05 smithi179 ceph-mon[63029]: pgmap v402: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:07.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.366+0000 7fa212600700 1 -- 172.21.15.84:0/440775555 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa20c1056c0 msgr2=0x7fa20c105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:07.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.366+0000 7fa212600700 1 --2- 172.21.15.84:0/440775555 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa20c1056c0 0x7fa20c105ae0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc0099c0 tx=0x7fa1fc009cd0 comp rx=0 tx=0).stop 2024-04-28T12:51:07.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.366+0000 7fa212600700 1 -- 172.21.15.84:0/440775555 shutdown_connections 2024-04-28T12:51:07.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.366+0000 7fa212600700 1 --2- 172.21.15.84:0/440775555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa20c107260 0x7fa20c10db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:07.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.366+0000 7fa212600700 1 --2- 172.21.15.84:0/440775555 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa20c1056c0 0x7fa20c105ae0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:07.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.366+0000 7fa212600700 1 --2- 172.21.15.84:0/440775555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa20c1068c0 0x7fa20c106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:07.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.366+0000 7fa212600700 1 -- 172.21.15.84:0/440775555 >> 172.21.15.84:0/440775555 conn(0x7fa20c100c80 msgr2=0x7fa20c1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:07.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.366+0000 7fa212600700 1 -- 172.21.15.84:0/440775555 shutdown_connections 2024-04-28T12:51:07.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.366+0000 7fa212600700 1 -- 172.21.15.84:0/440775555 wait complete. 2024-04-28T12:51:07.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.367+0000 7fa212600700 1 Processor -- start 2024-04-28T12:51:07.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.368+0000 7fa212600700 1 -- start start 2024-04-28T12:51:07.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.368+0000 7fa212600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa20c1056c0 0x7fa20c19b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:07.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.368+0000 7fa212600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa20c1068c0 0x7fa20c19bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:07.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.368+0000 7fa212600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa20c107260 0x7fa20c1a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:07.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.368+0000 7fa212600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fa20c111140 con 0x7fa20c1068c0 2024-04-28T12:51:07.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.368+0000 7fa212600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa20c110fc0 con 0x7fa20c1056c0 2024-04-28T12:51:07.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.368+0000 7fa212600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fa20c1112c0 con 0x7fa20c107260 2024-04-28T12:51:07.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.368+0000 7fa20b400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa20c1056c0 0x7fa20c19b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:07.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.368+0000 7fa20aa00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa20c1068c0 0x7fa20c19bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:07.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.368+0000 7fa20b400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa20c1056c0 0x7fa20c19b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:43502/0 (socket says 172.21.15.84:43502) 2024-04-28T12:51:07.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.368+0000 7fa20b400700 1 -- 172.21.15.84:0/4213861852 learned_addr learned my addr 172.21.15.84:0/4213861852 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:07.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.368+0000 7fa20be00700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa20c107260 0x7fa20c1a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:07.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.369+0000 7fa20b400700 1 -- 172.21.15.84:0/4213861852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa20c107260 msgr2=0x7fa20c1a2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:07.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.369+0000 7fa20b400700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa20c107260 0x7fa20c1a2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:07.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.369+0000 7fa20b400700 1 -- 172.21.15.84:0/4213861852 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa20c1068c0 msgr2=0x7fa20c19bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:07.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.369+0000 7fa20b400700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa20c1068c0 0x7fa20c19bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:07.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.369+0000 7fa20b400700 1 -- 172.21.15.84:0/4213861852 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20c1a3410 con 0x7fa20c1056c0 2024-04-28T12:51:07.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.369+0000 7fa20b400700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa20c1056c0 0x7fa20c19b7b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc003ee0 tx=0x7fa1fc003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:07.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.369+0000 7fa203e00700 1 -- 172.21.15.84:0/4213861852 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc009e60 con 0x7fa20c1056c0 2024-04-28T12:51:07.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.369+0000 7fa212600700 1 -- 172.21.15.84:0/4213861852 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa20c1a36a0 con 0x7fa20c1056c0 2024-04-28T12:51:07.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.370+0000 7fa212600700 1 -- 172.21.15.84:0/4213861852 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa20c1a3bd0 con 0x7fa20c1056c0 2024-04-28T12:51:07.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.370+0000 7fa203e00700 1 -- 172.21.15.84:0/4213861852 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1fc00f460 con 0x7fa20c1056c0 2024-04-28T12:51:07.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.370+0000 7fa203e00700 1 -- 172.21.15.84:0/4213861852 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc00bea0 con 0x7fa20c1056c0 2024-04-28T12:51:07.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.371+0000 7fa212600700 1 -- 172.21.15.84:0/4213861852 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1ec005360 con 0x7fa20c1056c0 2024-04-28T12:51:07.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.373+0000 7fa203e00700 1 -- 172.21.15.84:0/4213861852 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa1fc0052d0 con 0x7fa20c1056c0 2024-04-28T12:51:07.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.374+0000 7fa203e00700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa1f00a3cd0 0x7fa1f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:07.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.374+0000 7fa203e00700 1 -- 172.21.15.84:0/4213861852 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fa1fc0d1670 con 0x7fa20c1056c0 2024-04-28T12:51:07.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.375+0000 7fa20aa00700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa1f00a3cd0 0x7fa1f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:07.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.375+0000 7fa20aa00700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa1f00a3cd0 0x7fa1f00a6190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8005950 tx=0x7fa1f800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:07.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.377+0000 7fa203e00700 1 -- 172.21.15.84:0/4213861852 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa1fc013070 con 0x7fa20c1056c0 2024-04-28T12:51:07.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.608+0000 7fa212600700 1 -- 172.21.15.84:0/4213861852 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fa1ec000bf0 con 0x7fa1f00a3cd0 2024-04-28T12:51:07.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.611+0000 7fa203e00700 1 -- 172.21.15.84:0/4213861852 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fa1ec000bf0 con 0x7fa1f00a3cd0 2024-04-28T12:51:07.609 INFO:teuthology.orchestra.run.smithi084.stdout:HOST ADDR LABELS STATUS 2024-04-28T12:51:07.609 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084 172.21.15.84 2024-04-28T12:51:07.609 INFO:teuthology.orchestra.run.smithi084.stdout:smithi161 172.21.15.161 2024-04-28T12:51:07.609 INFO:teuthology.orchestra.run.smithi084.stdout:smithi179 172.21.15.179 2024-04-28T12:51:07.609 INFO:teuthology.orchestra.run.smithi084.stdout:3 hosts in cluster 2024-04-28T12:51:07.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.614+0000 7fa212600700 1 -- 172.21.15.84:0/4213861852 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa1f00a3cd0 msgr2=0x7fa1f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:07.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.614+0000 7fa212600700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa1f00a3cd0 0x7fa1f00a6190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8005950 tx=0x7fa1f800b410 comp rx=0 tx=0).stop 2024-04-28T12:51:07.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.614+0000 7fa212600700 1 -- 172.21.15.84:0/4213861852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa20c1056c0 msgr2=0x7fa20c19b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:07.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.614+0000 7fa212600700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa20c1056c0 0x7fa20c19b7b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc003ee0 tx=0x7fa1fc003f10 comp rx=0 tx=0).stop 2024-04-28T12:51:07.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.615+0000 7fa212600700 1 -- 172.21.15.84:0/4213861852 shutdown_connections 2024-04-28T12:51:07.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.615+0000 7fa212600700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fa1f00a3cd0 0x7fa1f00a6190 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:07.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.615+0000 7fa212600700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa20c1056c0 0x7fa20c19b7b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:07.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.615+0000 7fa212600700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa20c107260 0x7fa20c1a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:07.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.615+0000 7fa212600700 1 --2- 172.21.15.84:0/4213861852 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa20c1068c0 0x7fa20c19bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:07.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.615+0000 7fa212600700 1 -- 172.21.15.84:0/4213861852 >> 172.21.15.84:0/4213861852 conn(0x7fa20c100c80 msgr2=0x7fa20c109c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:07.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.615+0000 7fa212600700 1 -- 172.21.15.84:0/4213861852 shutdown_connections 2024-04-28T12:51:07.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:07.615+0000 7fa212600700 1 -- 172.21.15.84:0/4213861852 wait complete. 2024-04-28T12:51:07.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:07 smithi084 ceph-mon[28081]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:07.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:07 smithi084 ceph-mon[28081]: pgmap v403: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:07.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:07 smithi161 ceph-mon[62387]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:07.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:07 smithi161 ceph-mon[62387]: pgmap v403: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:08.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:07 smithi179 ceph-mon[63029]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:08.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:07 smithi179 ceph-mon[63029]: pgmap v403: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:08.604 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-28T12:51:09.054 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:51:09.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:09 smithi084 ceph-mon[28081]: pgmap v404: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:09.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:09 smithi161 ceph-mon[62387]: pgmap v404: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:10.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:09 smithi179 ceph-mon[63029]: pgmap v404: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:11.520 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.523+0000 7f3072e00700 1 -- 172.21.15.84:0/999032656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f306c109170 msgr2=0x7f306c10b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:11.521 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.523+0000 7f3072e00700 1 --2- 172.21.15.84:0/999032656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f306c109170 0x7f306c10b5e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f306400b0a0 tx=0x7f306400b3b0 comp rx=0 tx=0).stop 2024-04-28T12:51:11.521 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.523+0000 7f3072e00700 1 -- 172.21.15.84:0/999032656 shutdown_connections 2024-04-28T12:51:11.521 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.523+0000 7f3072e00700 1 --2- 172.21.15.84:0/999032656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f306c109170 0x7f306c10b5e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:11.521 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.523+0000 7f3072e00700 1 --2- 172.21.15.84:0/999032656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f306c103f10 0x7f306c106300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:11.521 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.523+0000 7f3072e00700 1 --2- 172.21.15.84:0/999032656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f306c106840 0x7f306c108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:11.521 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.523+0000 7f3072e00700 1 -- 172.21.15.84:0/999032656 >> 172.21.15.84:0/999032656 conn(0x7f306c0fd920 msgr2=0x7f306c0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:11.521 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.523+0000 7f3072e00700 1 -- 172.21.15.84:0/999032656 shutdown_connections 2024-04-28T12:51:11.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.523+0000 7f3072e00700 1 -- 172.21.15.84:0/999032656 wait complete. 2024-04-28T12:51:11.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.524+0000 7f3072e00700 1 Processor -- start 2024-04-28T12:51:11.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.524+0000 7f3072e00700 1 -- start start 2024-04-28T12:51:11.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.525+0000 7f3072e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f306c103f10 0x7f306c1995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:11.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.525+0000 7f3072e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f306c106840 0x7f306c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:11.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.525+0000 7f3072e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f306c109170 0x7f306c1a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:11.523 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.525+0000 7f3072e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f306c10eee0 con 0x7f306c106840 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.525+0000 7f3072e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f306c10ed60 con 0x7f306c109170 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.525+0000 7f3072e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f306c10f060 con 0x7f306c103f10 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.525+0000 7f306b400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f306c106840 0x7f306c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.525+0000 7f306b400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f306c106840 0x7f306c199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43182/0 (socket says 172.21.15.84:43182) 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.525+0000 7f306be00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f306c103f10 0x7f306c1995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.525+0000 7f306b400700 1 -- 172.21.15.84:0/53812847 learned_addr learned my addr 172.21.15.84:0/53812847 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.525+0000 7f3070e00700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f306c109170 0x7f306c1a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.526+0000 7f306b400700 1 -- 172.21.15.84:0/53812847 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f306c103f10 msgr2=0x7f306c1995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.526+0000 7f306b400700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f306c103f10 0x7f306c1995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.526+0000 7f306b400700 1 -- 172.21.15.84:0/53812847 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f306c109170 msgr2=0x7f306c1a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.526+0000 7f306b400700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f306c109170 0x7f306c1a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.526+0000 7f306b400700 1 -- 172.21.15.84:0/53812847 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f306c1a1210 con 0x7f306c106840 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.526+0000 7f306be00700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f306c103f10 0x7f306c1995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.526+0000 7f306b400700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f306c106840 0x7f306c199af0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f305800e9d0 tx=0x7f305800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:11.524 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.526+0000 7f3068c00700 1 -- 172.21.15.84:0/53812847 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305800cbd0 con 0x7f306c106840 2024-04-28T12:51:11.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.526+0000 7f3072e00700 1 -- 172.21.15.84:0/53812847 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f306c1a1500 con 0x7f306c106840 2024-04-28T12:51:11.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.526+0000 7f3068c00700 1 -- 172.21.15.84:0/53812847 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f305800ee70 con 0x7f306c106840 2024-04-28T12:51:11.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.526+0000 7f3068c00700 1 -- 172.21.15.84:0/53812847 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3058018680 con 0x7f306c106840 2024-04-28T12:51:11.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.527+0000 7f3072e00700 1 -- 172.21.15.84:0/53812847 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f306c1a1a90 con 0x7f306c106840 2024-04-28T12:51:11.525 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.528+0000 7f3072e00700 1 -- 172.21.15.84:0/53812847 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f306c06a110 con 0x7f306c106840 2024-04-28T12:51:11.527 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.530+0000 7f3068c00700 1 -- 172.21.15.84:0/53812847 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3058010430 con 0x7f306c106840 2024-04-28T12:51:11.528 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.532+0000 7f3068c00700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f30500a3cd0 0x7f30500a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:11.528 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.532+0000 7f3068c00700 1 -- 172.21.15.84:0/53812847 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f30580d0ba0 con 0x7f306c106840 2024-04-28T12:51:11.529 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.532+0000 7f306be00700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f30500a3cd0 0x7f30500a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:11.529 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.533+0000 7f306be00700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f30500a3cd0 0x7f30500a6190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f305c011490 tx=0x7f305c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:11.538 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.541+0000 7f3068c00700 1 -- 172.21.15.84:0/53812847 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f305809d040 con 0x7f306c106840 2024-04-28T12:51:11.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.780+0000 7f3072e00700 1 -- 172.21.15.84:0/53812847 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f306c063860 con 0x7f30500a3cd0 2024-04-28T12:51:11.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.783+0000 7f3068c00700 1 -- 172.21.15.84:0/53812847 <== mgr.14152 v2:172.21.15.84:6800/2396287759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f306c063860 con 0x7f30500a3cd0 2024-04-28T12:51:11.780 INFO:teuthology.orchestra.run.smithi084.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-28T12:51:11.780 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400WF400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-28T12:51:11.780 INFO:teuthology.orchestra.run.smithi084.stdout:smithi161 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233001X400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-28T12:51:11.780 INFO:teuthology.orchestra.run.smithi084.stdout:smithi179 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6303009J400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-28T12:51:11.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.786+0000 7f3072e00700 1 -- 172.21.15.84:0/53812847 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f30500a3cd0 msgr2=0x7f30500a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:11.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.786+0000 7f3072e00700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f30500a3cd0 0x7f30500a6190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f305c011490 tx=0x7f305c009260 comp rx=0 tx=0).stop 2024-04-28T12:51:11.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.786+0000 7f3072e00700 1 -- 172.21.15.84:0/53812847 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f306c106840 msgr2=0x7f306c199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:11.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.786+0000 7f3072e00700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f306c106840 0x7f306c199af0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f305800e9d0 tx=0x7f305800ece0 comp rx=0 tx=0).stop 2024-04-28T12:51:11.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.786+0000 7f3072e00700 1 -- 172.21.15.84:0/53812847 shutdown_connections 2024-04-28T12:51:11.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.786+0000 7f3072e00700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f30500a3cd0 0x7f30500a6190 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:11.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.786+0000 7f3072e00700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f306c109170 0x7f306c1a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:11.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.786+0000 7f3072e00700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f306c103f10 0x7f306c1995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:11.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.786+0000 7f3072e00700 1 --2- 172.21.15.84:0/53812847 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f306c106840 0x7f306c199af0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:11.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.786+0000 7f3072e00700 1 -- 172.21.15.84:0/53812847 >> 172.21.15.84:0/53812847 conn(0x7f306c0fd920 msgr2=0x7f306c104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:11.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.786+0000 7f3072e00700 1 -- 172.21.15.84:0/53812847 shutdown_connections 2024-04-28T12:51:11.784 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:11.786+0000 7f3072e00700 1 -- 172.21.15.84:0/53812847 wait complete. 2024-04-28T12:51:11.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:11 smithi161 ceph-mon[62387]: pgmap v405: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:12.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:11 smithi179 ceph-mon[63029]: pgmap v405: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:12.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:11 smithi084 ceph-mon[28081]: pgmap v405: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:12.666 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-28T12:51:12.678 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi084.front.sepia.ceph.com 2024-04-28T12:51:12.678 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-28T12:51:12.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:12 smithi084 ceph-mon[28081]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:13.103 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:51:13.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:12 smithi179 ceph-mon[63029]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:13.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:12 smithi161 ceph-mon[62387]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:13.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:13 smithi084 ceph-mon[28081]: pgmap v406: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:14.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:13 smithi179 ceph-mon[63029]: pgmap v406: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:13 smithi161 ceph-mon[62387]: pgmap v406: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:15.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.586+0000 7f104d400700 1 -- 172.21.15.84:0/855824123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1048107160 msgr2=0x7f104810da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:15.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.586+0000 7f104d400700 1 --2- 172.21.15.84:0/855824123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1048107160 0x7f104810da10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f104000b0a0 tx=0x7f104000b3b0 comp rx=0 tx=0).stop 2024-04-28T12:51:15.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.586+0000 7f104d400700 1 -- 172.21.15.84:0/855824123 shutdown_connections 2024-04-28T12:51:15.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.586+0000 7f104d400700 1 --2- 172.21.15.84:0/855824123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10481055c0 0x7f10481059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:15.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.586+0000 7f104d400700 1 --2- 172.21.15.84:0/855824123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1048107160 0x7f104810da10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:15.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.586+0000 7f104d400700 1 --2- 172.21.15.84:0/855824123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10481067c0 0x7f1048106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:15.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.586+0000 7f104d400700 1 -- 172.21.15.84:0/855824123 >> 172.21.15.84:0/855824123 conn(0x7f1048100b60 msgr2=0x7f1048102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:15.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.586+0000 7f104d400700 1 -- 172.21.15.84:0/855824123 shutdown_connections 2024-04-28T12:51:15.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.586+0000 7f104d400700 1 -- 172.21.15.84:0/855824123 wait complete. 2024-04-28T12:51:15.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.587+0000 7f104d400700 1 Processor -- start 2024-04-28T12:51:15.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.587+0000 7f104d400700 1 -- start start 2024-04-28T12:51:15.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.587+0000 7f104d400700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10481055c0 0x7f104819b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:15.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f104d400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10481067c0 0x7f104819bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:15.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f104d400700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1048107160 0x7f10481a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:15.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f104d400700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1048111050 con 0x7f1048107160 2024-04-28T12:51:15.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f104d400700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1048110ed0 con 0x7f10481067c0 2024-04-28T12:51:15.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f104d400700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f10481111d0 con 0x7f10481055c0 2024-04-28T12:51:15.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f104ca00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1048107160 0x7f10481a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:15.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f1047e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10481055c0 0x7f104819b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:15.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f104ca00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1048107160 0x7f10481a2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43204/0 (socket says 172.21.15.84:43204) 2024-04-28T12:51:15.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f104ca00700 1 -- 172.21.15.84:0/1386508924 learned_addr learned my addr 172.21.15.84:0/1386508924 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:15.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f1047400700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10481067c0 0x7f104819bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:15.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f1047e00700 1 -- 172.21.15.84:0/1386508924 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10481067c0 msgr2=0x7f104819bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:15.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f1047e00700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10481067c0 0x7f104819bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:15.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f1047e00700 1 -- 172.21.15.84:0/1386508924 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1048107160 msgr2=0x7f10481a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:15.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f1047e00700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1048107160 0x7f10481a2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:15.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f1047e00700 1 -- 172.21.15.84:0/1386508924 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10481a32a0 con 0x7f10481055c0 2024-04-28T12:51:15.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.588+0000 7f104ca00700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1048107160 0x7f10481a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:15.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.589+0000 7f1047e00700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10481055c0 0x7f104819b640 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f103800e9f0 tx=0x7f103800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:15.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.589+0000 7f1044c00700 1 -- 172.21.15.84:0/1386508924 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038004230 con 0x7f10481055c0 2024-04-28T12:51:15.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.589+0000 7f1044c00700 1 -- 172.21.15.84:0/1386508924 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10380043d0 con 0x7f10481055c0 2024-04-28T12:51:15.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.589+0000 7f1044c00700 1 -- 172.21.15.84:0/1386508924 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038003d30 con 0x7f10481055c0 2024-04-28T12:51:15.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.589+0000 7f104d400700 1 -- 172.21.15.84:0/1386508924 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10481a3590 con 0x7f10481055c0 2024-04-28T12:51:15.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.589+0000 7f104d400700 1 -- 172.21.15.84:0/1386508924 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10481a3ad0 con 0x7f10481055c0 2024-04-28T12:51:15.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.591+0000 7f104d400700 1 -- 172.21.15.84:0/1386508924 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f104806a110 con 0x7f10481055c0 2024-04-28T12:51:15.589 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.592+0000 7f1044c00700 1 -- 172.21.15.84:0/1386508924 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f103800c7c0 con 0x7f10481055c0 2024-04-28T12:51:15.590 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.594+0000 7f1044c00700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f102c0a3d50 0x7f102c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:15.590 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.594+0000 7f1044c00700 1 -- 172.21.15.84:0/1386508924 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f10380d0c20 con 0x7f10481055c0 2024-04-28T12:51:15.590 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.594+0000 7f1047400700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f102c0a3d50 0x7f102c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:15.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.595+0000 7f1047400700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f102c0a3d50 0x7f102c0a6210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1034009c60 tx=0x7f1034009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:15.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.598+0000 7f1044c00700 1 -- 172.21.15.84:0/1386508924 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f103809d0c0 con 0x7f10481055c0 2024-04-28T12:51:15.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.874+0000 7f104d400700 1 -- 172.21.15.84:0/1386508924 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f104802e490 con 0x7f10481055c0 2024-04-28T12:51:15.872 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:15 smithi084 ceph-mon[28081]: pgmap v407: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:15.872 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.875+0000 7f1044c00700 1 -- 172.21.15.84:0/1386508924 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3060 (secure 0 0 0) 0x7f103809c9c0 con 0x7f10481055c0 2024-04-28T12:51:15.874 INFO:teuthology.orchestra.run.smithi084.stdout:e24 2024-04-28T12:51:15.874 INFO:teuthology.orchestra.run.smithi084.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-28T12:51:15.874 INFO:teuthology.orchestra.run.smithi084.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T12:51:15.874 INFO:teuthology.orchestra.run.smithi084.stdout:legacy client fscid: 1 2024-04-28T12:51:15.874 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:51:15.874 INFO:teuthology.orchestra.run.smithi084.stdout:Filesystem 'cephfs' (1) 2024-04-28T12:51:15.874 INFO:teuthology.orchestra.run.smithi084.stdout:fs_name cephfs 2024-04-28T12:51:15.874 INFO:teuthology.orchestra.run.smithi084.stdout:epoch 24 2024-04-28T12:51:15.874 INFO:teuthology.orchestra.run.smithi084.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-28T12:51:15.874 INFO:teuthology.orchestra.run.smithi084.stdout:created 2024-04-28T12:48:34.334879+0000 2024-04-28T12:51:15.874 INFO:teuthology.orchestra.run.smithi084.stdout:modified 2024-04-28T12:49:08.738419+0000 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:tableserver 0 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:root 0 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:session_timeout 300 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:session_autoclose 300 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:max_file_size 1099511627776 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:max_xattr_size 65536 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:required_client_features {} 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:last_failure 0 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:last_failure_osd_epoch 0 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:max_mds 3 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:in 0,1,2 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:up {0=24602,1=24562,2=24545} 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:failed 2024-04-28T12:51:15.875 INFO:teuthology.orchestra.run.smithi084.stdout:damaged 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:stopped 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:data_pools [3,4] 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:metadata_pool 2 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:inline_data disabled 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:balancer 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:bal_rank_mask -1 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:standby_count_wanted 0 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:qdb_cluster leader: 24602 members: 24545,24562,24602 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.b{0:24602} state up:active seq 3 addr [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.i{0:24544} state up:standby-replay seq 1 addr [v2:172.21.15.179:6838/3879704230,v1:172.21.15.179:6839/3879704230] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.g{1:24562} state up:active seq 13 addr [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.c{1:24520} state up:standby-replay seq 1 addr [v2:172.21.15.179:6836/2376856149,v1:172.21.15.179:6837/2376856149] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.h{2:24545} state up:active seq 15 addr [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.e{2:24509} state up:standby-replay seq 1 addr [v2:172.21.15.161:6834/3348651842,v1:172.21.15.161:6835/3348651842] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:51:15.876 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:51:15.877 INFO:teuthology.orchestra.run.smithi084.stdout:Standby daemons: 2024-04-28T12:51:15.877 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:51:15.877 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.j{-1:14385} state up:standby seq 1 addr [v2:172.21.15.84:6834/3549575919,v1:172.21.15.84:6835/3549575919] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:51:15.877 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.d{-1:14397} state up:standby seq 1 addr [v2:172.21.15.84:6836/2455857863,v1:172.21.15.84:6837/2455857863] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:51:15.877 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.a{-1:14469} state up:standby seq 1 addr [v2:172.21.15.84:6840/1196924806,v1:172.21.15.84:6841/1196924806] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:51:15.877 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.l{-1:24442} state up:standby seq 1 addr [v2:172.21.15.179:6832/2409757407,v1:172.21.15.179:6833/2409757407] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:51:15.877 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.k{-1:24461} state up:standby seq 1 addr [v2:172.21.15.161:6832/3034002883,v1:172.21.15.161:6833/3034002883] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:51:15.877 INFO:teuthology.orchestra.run.smithi084.stdout:[mds.f{-1:24508} state up:standby seq 1 addr [v2:172.21.15.179:6834/476402611,v1:172.21.15.179:6835/476402611] compat {c=[1],r=[1],i=[1fff]}] 2024-04-28T12:51:15.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.879+0000 7f104d400700 1 -- 172.21.15.84:0/1386508924 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f102c0a3d50 msgr2=0x7f102c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:15.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.879+0000 7f104d400700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f102c0a3d50 0x7f102c0a6210 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1034009c60 tx=0x7f1034009260 comp rx=0 tx=0).stop 2024-04-28T12:51:15.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.879+0000 7f104d400700 1 -- 172.21.15.84:0/1386508924 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10481055c0 msgr2=0x7f104819b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:15.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.879+0000 7f104d400700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10481055c0 0x7f104819b640 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f103800e9f0 tx=0x7f103800ed00 comp rx=0 tx=0).stop 2024-04-28T12:51:15.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.879+0000 7f104d400700 1 -- 172.21.15.84:0/1386508924 shutdown_connections 2024-04-28T12:51:15.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.879+0000 7f104d400700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1048107160 0x7f10481a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:15.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.879+0000 7f104d400700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10481055c0 0x7f104819b640 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:15.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.879+0000 7f104d400700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f102c0a3d50 0x7f102c0a6210 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:15.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.879+0000 7f104d400700 1 --2- 172.21.15.84:0/1386508924 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10481067c0 0x7f104819bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:15.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.879+0000 7f104d400700 1 -- 172.21.15.84:0/1386508924 >> 172.21.15.84:0/1386508924 conn(0x7f1048100b60 msgr2=0x7f1048109ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:15.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.879+0000 7f104d400700 1 -- 172.21.15.84:0/1386508924 shutdown_connections 2024-04-28T12:51:15.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:15.879+0000 7f104d400700 1 -- 172.21.15.84:0/1386508924 wait complete. 2024-04-28T12:51:15.879 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 24 2024-04-28T12:51:15.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:15 smithi161 ceph-mon[62387]: pgmap v407: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:16.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:15 smithi179 ceph-mon[63029]: pgmap v407: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:16.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-28T12:51:17.067 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:16 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1386508924' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-28T12:51:17.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:16 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1386508924' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-28T12:51:17.173 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:51:17.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:16 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1386508924' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-28T12:51:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:17 smithi179 ceph-mon[63029]: pgmap v408: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:18.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:17 smithi084 ceph-mon[28081]: pgmap v408: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:18.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:17 smithi161 ceph-mon[62387]: pgmap v408: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:19.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.551+0000 7f7ee3600700 1 -- 172.21.15.84:0/2728551196 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7edc10b350 msgr2=0x7f7edc10d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:19.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.551+0000 7f7ee3600700 1 --2- 172.21.15.84:0/2728551196 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7edc10b350 0x7f7edc10d7e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400b0a0 tx=0x7f7ed400b3b0 comp rx=0 tx=0).stop 2024-04-28T12:51:19.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.551+0000 7f7ee3600700 1 -- 172.21.15.84:0/2728551196 shutdown_connections 2024-04-28T12:51:19.548 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.551+0000 7f7ee3600700 1 --2- 172.21.15.84:0/2728551196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7edc077450 0x7f7edc0758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:19.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.551+0000 7f7ee3600700 1 --2- 172.21.15.84:0/2728551196 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7edc10b350 0x7f7edc10d7e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:19.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.551+0000 7f7ee3600700 1 --2- 172.21.15.84:0/2728551196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7edc075e80 0x7f7edc076300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:19.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.551+0000 7f7ee3600700 1 -- 172.21.15.84:0/2728551196 >> 172.21.15.84:0/2728551196 conn(0x7f7edc0fea20 msgr2=0x7f7edc100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:19.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.551+0000 7f7ee3600700 1 -- 172.21.15.84:0/2728551196 shutdown_connections 2024-04-28T12:51:19.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.552+0000 7f7ee3600700 1 -- 172.21.15.84:0/2728551196 wait complete. 2024-04-28T12:51:19.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.553+0000 7f7ee3600700 1 Processor -- start 2024-04-28T12:51:19.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.553+0000 7f7ee3600700 1 -- start start 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.553+0000 7f7ee3600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7edc075e80 0x7f7edc19b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.553+0000 7f7ee3600700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7edc077450 0x7f7edc19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.553+0000 7f7ee3600700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7edc10b350 0x7f7edc1a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.553+0000 7f7ee3600700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc111190 con 0x7f7edc10b350 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.553+0000 7f7ee3600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc111010 con 0x7f7edc075e80 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.553+0000 7f7ee3600700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc111310 con 0x7f7edc077450 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.553+0000 7f7ee0c00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7edc075e80 0x7f7edc19b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.554+0000 7f7ee0c00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7edc075e80 0x7f7edc19b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58636/0 (socket says 172.21.15.84:58636) 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.554+0000 7f7ee0c00700 1 -- 172.21.15.84:0/1051275083 learned_addr learned my addr 172.21.15.84:0/1051275083 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.554+0000 7f7edbe00700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7edc077450 0x7f7edc19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.554+0000 7f7ee1600700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7edc10b350 0x7f7edc1a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.554+0000 7f7ee0c00700 1 -- 172.21.15.84:0/1051275083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7edc077450 msgr2=0x7f7edc19bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.554+0000 7f7ee0c00700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7edc077450 0x7f7edc19bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.554+0000 7f7ee0c00700 1 -- 172.21.15.84:0/1051275083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7edc10b350 msgr2=0x7f7edc1a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:19.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.554+0000 7f7ee0c00700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7edc10b350 0x7f7edc1a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:19.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.554+0000 7f7ee0c00700 1 -- 172.21.15.84:0/1051275083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7edc1a3450 con 0x7f7edc075e80 2024-04-28T12:51:19.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.554+0000 7f7edbe00700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7edc077450 0x7f7edc19bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:19.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.554+0000 7f7ee0c00700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7edc075e80 0x7f7edc19b7f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7ed000d950 tx=0x7f7ed000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:19.553 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.555+0000 7f7ed9600700 1 -- 172.21.15.84:0/1051275083 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0016070 con 0x7f7edc075e80 2024-04-28T12:51:19.553 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.555+0000 7f7ee3600700 1 -- 172.21.15.84:0/1051275083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7edc1a3740 con 0x7f7edc075e80 2024-04-28T12:51:19.553 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.555+0000 7f7ed9600700 1 -- 172.21.15.84:0/1051275083 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ed0004830 con 0x7f7edc075e80 2024-04-28T12:51:19.553 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.555+0000 7f7ed9600700 1 -- 172.21.15.84:0/1051275083 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0005520 con 0x7f7edc075e80 2024-04-28T12:51:19.553 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.555+0000 7f7ee3600700 1 -- 172.21.15.84:0/1051275083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7edc1a3cd0 con 0x7f7edc075e80 2024-04-28T12:51:19.553 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.556+0000 7f7ee3600700 1 -- 172.21.15.84:0/1051275083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7edc06a110 con 0x7f7edc075e80 2024-04-28T12:51:19.554 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.558+0000 7f7ed9600700 1 -- 172.21.15.84:0/1051275083 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7ed0004220 con 0x7f7edc075e80 2024-04-28T12:51:19.556 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.560+0000 7f7ed9600700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ec40a3d50 0x7f7ec40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:19.556 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.560+0000 7f7ed9600700 1 -- 172.21.15.84:0/1051275083 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f7ed00d05d0 con 0x7f7edc075e80 2024-04-28T12:51:19.556 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.560+0000 7f7edbe00700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ec40a3d50 0x7f7ec40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:19.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.560+0000 7f7edbe00700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ec40a3d50 0x7f7ec40a6210 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7edc077180 tx=0x7f7ec8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:19.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.564+0000 7f7ed9600700 1 -- 172.21.15.84:0/1051275083 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7ed001f020 con 0x7f7edc075e80 2024-04-28T12:51:19.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.798+0000 7f7ee3600700 1 -- 172.21.15.84:0/1051275083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f7edc064030 con 0x7f7edc075e80 2024-04-28T12:51:19.797 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.800+0000 7f7ed9600700 1 -- 172.21.15.84:0/1051275083 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v94) ==== 56+0+5624 (secure 0 0 0) 0x7f7ed009ca70 con 0x7f7edc075e80 2024-04-28T12:51:19.797 INFO:teuthology.orchestra.run.smithi084.stdout:epoch 94 2024-04-28T12:51:19.798 INFO:teuthology.orchestra.run.smithi084.stdout:fsid ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T12:51:19.798 INFO:teuthology.orchestra.run.smithi084.stdout:created 2024-04-28T12:39:22.248672+0000 2024-04-28T12:51:19.798 INFO:teuthology.orchestra.run.smithi084.stdout:modified 2024-04-28T12:50:30.848698+0000 2024-04-28T12:51:19.798 INFO:teuthology.orchestra.run.smithi084.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-28T12:51:19.798 INFO:teuthology.orchestra.run.smithi084.stdout:crush_version 27 2024-04-28T12:51:19.798 INFO:teuthology.orchestra.run.smithi084.stdout:full_ratio 0.95 2024-04-28T12:51:19.798 INFO:teuthology.orchestra.run.smithi084.stdout:backfillfull_ratio 0.9 2024-04-28T12:51:19.799 INFO:teuthology.orchestra.run.smithi084.stdout:nearfull_ratio 0.85 2024-04-28T12:51:19.799 INFO:teuthology.orchestra.run.smithi084.stdout:require_min_compat_client luminous 2024-04-28T12:51:19.799 INFO:teuthology.orchestra.run.smithi084.stdout:min_compat_client luminous 2024-04-28T12:51:19.799 INFO:teuthology.orchestra.run.smithi084.stdout:require_osd_release squid 2024-04-28T12:51:19.799 INFO:teuthology.orchestra.run.smithi084.stdout:stretch_mode_enabled false 2024-04-28T12:51:19.799 INFO:teuthology.orchestra.run.smithi084.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-28T12:51:19.799 INFO:teuthology.orchestra.run.smithi084.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-28T12:51:19.799 INFO:teuthology.orchestra.run.smithi084.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-28T12:51:19.799 INFO:teuthology.orchestra.run.smithi084.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-28T12:51:19.799 INFO:teuthology.orchestra.run.smithi084.stdout:max_osd 12 2024-04-28T12:51:19.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0 up in weight 1 up_from 8 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.84:6802/598031287,v1:172.21.15.84:6803/598031287] [v2:172.21.15.84:6804/598031287,v1:172.21.15.84:6805/598031287] exists,up 2099232c-3140-41c4-83b4-89da6fa3a2fb 2024-04-28T12:51:19.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1 up in weight 1 up_from 13 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6800/449593633,v1:172.21.15.161:6801/449593633] [v2:172.21.15.161:6802/449593633,v1:172.21.15.161:6803/449593633] exists,up ee0ea48d-5933-4d73-a6ee-a9028badef12 2024-04-28T12:51:19.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2 up in weight 1 up_from 18 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6800/3974191146,v1:172.21.15.179:6801/3974191146] [v2:172.21.15.179:6802/3974191146,v1:172.21.15.179:6803/3974191146] exists,up ac1f166c-e9a5-483c-9551-026ed03d1564 2024-04-28T12:51:19.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 up in weight 1 up_from 25 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.84:6810/3690923628,v1:172.21.15.84:6811/3690923628] [v2:172.21.15.84:6812/3690923628,v1:172.21.15.84:6813/3690923628] exists,up 547c4122-2dd9-4a63-a484-f615f99b428c 2024-04-28T12:51:19.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4 up in weight 1 up_from 33 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6808/1673806515,v1:172.21.15.161:6809/1673806515] [v2:172.21.15.161:6810/1673806515,v1:172.21.15.161:6811/1673806515] exists,up dd99b566-f29a-42c6-b04d-7122b6787464 2024-04-28T12:51:19.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5 up in weight 1 up_from 38 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6808/2387861251,v1:172.21.15.179:6809/2387861251] [v2:172.21.15.179:6810/2387861251,v1:172.21.15.179:6811/2387861251] exists,up e0697f1f-142e-43d7-b718-73f40970276e 2024-04-28T12:51:19.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6 up in weight 1 up_from 46 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.84:6818/440861785,v1:172.21.15.84:6819/440861785] [v2:172.21.15.84:6820/440861785,v1:172.21.15.84:6821/440861785] exists,up 1057554e-494f-4f9e-9823-a9e63c483a5f 2024-04-28T12:51:19.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7 up in weight 1 up_from 51 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6816/1226878087,v1:172.21.15.161:6817/1226878087] [v2:172.21.15.161:6818/1226878087,v1:172.21.15.161:6819/1226878087] exists,up 7108aa59-cf9a-4903-89fa-d0c456f0ed1d 2024-04-28T12:51:19.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.8 up in weight 1 up_from 59 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6816/277418948,v1:172.21.15.179:6817/277418948] [v2:172.21.15.179:6818/277418948,v1:172.21.15.179:6819/277418948] exists,up 0b005d60-667b-4061-80e9-af319ce052df 2024-04-28T12:51:19.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.9 up in weight 1 up_from 67 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.84:6826/3448177312,v1:172.21.15.84:6827/3448177312] [v2:172.21.15.84:6828/3448177312,v1:172.21.15.84:6829/3448177312] exists,up 7bc308bb-cc85-40b4-9af0-7dec273748f7 2024-04-28T12:51:19.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.10 up in weight 1 up_from 72 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6824/3240597178,v1:172.21.15.161:6825/3240597178] [v2:172.21.15.161:6826/3240597178,v1:172.21.15.161:6827/3240597178] exists,up 3f25ff51-6d47-4313-a7af-4be37046c627 2024-04-28T12:51:19.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.11 up in weight 1 up_from 80 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6824/3474527890,v1:172.21.15.179:6825/3474527890] [v2:172.21.15.179:6826/3474527890,v1:172.21.15.179:6827/3474527890] exists,up 6babaf1e-dfe0-4f33-a484-ba66a566d5d3 2024-04-28T12:51:19.800 INFO:teuthology.orchestra.run.smithi084.stdout:pg_upmap_items 2.4 [2,11] 2024-04-28T12:51:19.800 INFO:teuthology.orchestra.run.smithi084.stdout:pg_upmap_items 2.6 [8,11] 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:pg_upmap_items 3.e [4,6] 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:pg_upmap_items 3.10 [5,9] 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:pg_upmap_items 3.11 [4,8] 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:pg_upmap_items 3.18 [3,9] 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:pg_upmap_items 3.2c [4,9] 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:pg_upmap_items 4.6 [9,11] 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:pg_upmap_items 4.8 [5,10] 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:pg_upmap_items 4.b [9,8] 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:pg_upmap_items 4.13 [9,8] 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/1611938555 expires 2024-04-29T12:40:29.445750+0000 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6801/4213070051 expires 2024-04-29T12:40:29.445750+0000 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6800/4213070051 expires 2024-04-29T12:40:29.445750+0000 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/173746096 expires 2024-04-29T12:40:29.445750+0000 2024-04-28T12:51:19.801 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/3031983413 expires 2024-04-29T12:39:55.311912+0000 2024-04-28T12:51:19.802 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/3332561455 expires 2024-04-29T12:39:55.311912+0000 2024-04-28T12:51:19.802 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/3243457783 expires 2024-04-29T12:39:55.311912+0000 2024-04-28T12:51:19.802 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6800/4148140420 expires 2024-04-29T12:39:55.311912+0000 2024-04-28T12:51:19.802 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/3787967310 expires 2024-04-29T12:40:29.445750+0000 2024-04-28T12:51:19.802 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6801/4148140420 expires 2024-04-29T12:39:55.311912+0000 2024-04-28T12:51:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.803+0000 7f7ee3600700 1 -- 172.21.15.84:0/1051275083 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ec40a3d50 msgr2=0x7f7ec40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.803+0000 7f7ee3600700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ec40a3d50 0x7f7ec40a6210 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7edc077180 tx=0x7f7ec8009260 comp rx=0 tx=0).stop 2024-04-28T12:51:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.803+0000 7f7ee3600700 1 -- 172.21.15.84:0/1051275083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7edc075e80 msgr2=0x7f7edc19b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.803+0000 7f7ee3600700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7edc075e80 0x7f7edc19b7f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7ed000d950 tx=0x7f7ed000dc60 comp rx=0 tx=0).stop 2024-04-28T12:51:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.804+0000 7f7ee3600700 1 -- 172.21.15.84:0/1051275083 shutdown_connections 2024-04-28T12:51:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.804+0000 7f7ee3600700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ec40a3d50 0x7f7ec40a6210 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.804+0000 7f7ee3600700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7edc075e80 0x7f7edc19b7f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.804+0000 7f7ee3600700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7edc10b350 0x7f7edc1a2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.804+0000 7f7ee3600700 1 --2- 172.21.15.84:0/1051275083 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7edc077450 0x7f7edc19bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.804+0000 7f7ee3600700 1 -- 172.21.15.84:0/1051275083 >> 172.21.15.84:0/1051275083 conn(0x7f7edc0fea20 msgr2=0x7f7edc107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:19.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.804+0000 7f7ee3600700 1 -- 172.21.15.84:0/1051275083 shutdown_connections 2024-04-28T12:51:19.804 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:19.804+0000 7f7ee3600700 1 -- 172.21.15.84:0/1051275083 wait complete. 2024-04-28T12:51:20.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:19 smithi179 ceph-mon[63029]: pgmap v409: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:20.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:19 smithi084 ceph-mon[28081]: pgmap v409: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:20.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:19 smithi161 ceph-mon[62387]: pgmap v409: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:20.851 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-28T12:51:20.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:51:21.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.073+0000 7ff68b800640 1 -- 172.21.15.84:0/2359552008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff684108f90 msgr2=0x7ff6841093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.073+0000 7ff68b800640 1 --2- 172.21.15.84:0/2359552008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff684108f90 0x7ff6841093b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff674009900 tx=0x7ff67402a400 comp rx=0 tx=0).stop 2024-04-28T12:51:21.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.073+0000 7ff68b800640 1 -- 172.21.15.84:0/2359552008 shutdown_connections 2024-04-28T12:51:21.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.073+0000 7ff68b800640 1 --2- 172.21.15.84:0/2359552008 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff68410ab40 0x7ff6841113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.073+0000 7ff68b800640 1 --2- 172.21.15.84:0/2359552008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff68410a180 0x7ff68410a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.073+0000 7ff68b800640 1 --2- 172.21.15.84:0/2359552008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff684108f90 0x7ff6841093b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.073+0000 7ff68b800640 1 -- 172.21.15.84:0/2359552008 >> 172.21.15.84:0/2359552008 conn(0x7ff684077f40 msgr2=0x7ff68407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:21.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.073+0000 7ff68b800640 1 -- 172.21.15.84:0/2359552008 shutdown_connections 2024-04-28T12:51:21.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.073+0000 7ff68b800640 1 -- 172.21.15.84:0/2359552008 wait complete. 2024-04-28T12:51:21.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.074+0000 7ff68b800640 1 Processor -- start 2024-04-28T12:51:21.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff68b800640 1 -- start start 2024-04-28T12:51:21.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff68b800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff684108f90 0x7ff68419f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:21.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff68b800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff68410a180 0x7ff68419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:21.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff68b800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff68410ab40 0x7ff6841a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:21.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff68b800640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff684114b20 con 0x7ff68410ab40 2024-04-28T12:51:21.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff68b800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff6841149a0 con 0x7ff68410a180 2024-04-28T12:51:21.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff68b800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff684114ca0 con 0x7ff684108f90 2024-04-28T12:51:21.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff689800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff68410ab40 0x7ff6841a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff688e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff684108f90 0x7ff68419f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff689800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff68410ab40 0x7ff6841a6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42912/0 (socket says 172.21.15.84:42912) 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff688e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff684108f90 0x7ff68419f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:44022/0 (socket says 172.21.15.84:44022) 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff689800640 1 -- 172.21.15.84:0/3666067459 learned_addr learned my addr 172.21.15.84:0/3666067459 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.075+0000 7ff67be00640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff68410a180 0x7ff68419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.076+0000 7ff688e00640 1 -- 172.21.15.84:0/3666067459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff68410a180 msgr2=0x7ff68419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.076+0000 7ff688e00640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff68410a180 0x7ff68419f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.076+0000 7ff688e00640 1 -- 172.21.15.84:0/3666067459 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff68410ab40 msgr2=0x7ff6841a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.076+0000 7ff688e00640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff68410ab40 0x7ff6841a6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.076+0000 7ff688e00640 1 -- 172.21.15.84:0/3666067459 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6841a6df0 con 0x7ff684108f90 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.076+0000 7ff688e00640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff684108f90 0x7ff68419f1d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff67402a920 tx=0x7ff674003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.076+0000 7ff682000640 1 -- 172.21.15.84:0/3666067459 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67403f070 con 0x7ff684108f90 2024-04-28T12:51:21.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.076+0000 7ff682000640 1 -- 172.21.15.84:0/3666067459 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff674031c50 con 0x7ff684108f90 2024-04-28T12:51:21.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.076+0000 7ff682000640 1 -- 172.21.15.84:0/3666067459 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff674035030 con 0x7ff684108f90 2024-04-28T12:51:21.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.076+0000 7ff68b800640 1 -- 172.21.15.84:0/3666067459 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6841a7080 con 0x7ff684108f90 2024-04-28T12:51:21.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.076+0000 7ff68b800640 1 -- 172.21.15.84:0/3666067459 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6841a74f0 con 0x7ff684108f90 2024-04-28T12:51:21.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.077+0000 7ff68b800640 1 -- 172.21.15.84:0/3666067459 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff684110380 con 0x7ff684108f90 2024-04-28T12:51:21.076 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.080+0000 7ff682000640 1 -- 172.21.15.84:0/3666067459 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff674042070 con 0x7ff684108f90 2024-04-28T12:51:21.077 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.081+0000 7ff682000640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff6680a3cc0 0x7ff6680a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:21.077 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.081+0000 7ff682000640 1 -- 172.21.15.84:0/3666067459 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff6740f1960 con 0x7ff684108f90 2024-04-28T12:51:21.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.081+0000 7ff67be00640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff6680a3cc0 0x7ff6680a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:21.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.082+0000 7ff67be00640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff6680a3cc0 0x7ff6680a6180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff6841a06e0 tx=0x7ff66c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:21.079 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.083+0000 7ff682000640 1 -- 172.21.15.84:0/3666067459 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff6740bd7b0 con 0x7ff684108f90 2024-04-28T12:51:21.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:20 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1051275083' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-28T12:51:21.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:20 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1051275083' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-28T12:51:21.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:20 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1051275083' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-28T12:51:21.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.286+0000 7ff68b800640 1 -- 172.21.15.84:0/3666067459 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff68402dca0 con 0x7ff684108f90 2024-04-28T12:51:21.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.288+0000 7ff682000640 1 -- 172.21.15.84:0/3666067459 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24634 (secure 0 0 0) 0x7ff6740c1ab0 con 0x7ff684108f90 2024-04-28T12:51:21.286 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:51:21.286 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":94,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:50:30.848698+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.8","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:51:21.288 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.291+0000 7ff68b800640 1 -- 172.21.15.84:0/3666067459 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff6680a3cc0 msgr2=0x7ff6680a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.288 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.291+0000 7ff68b800640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff6680a3cc0 0x7ff6680a6180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff6841a06e0 tx=0x7ff66c005f90 comp rx=0 tx=0).stop 2024-04-28T12:51:21.288 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.291+0000 7ff68b800640 1 -- 172.21.15.84:0/3666067459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff684108f90 msgr2=0x7ff68419f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.291+0000 7ff68b800640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff684108f90 0x7ff68419f1d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff67402a920 tx=0x7ff674003c10 comp rx=0 tx=0).stop 2024-04-28T12:51:21.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.292+0000 7ff68b800640 1 -- 172.21.15.84:0/3666067459 shutdown_connections 2024-04-28T12:51:21.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.292+0000 7ff68b800640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff6680a3cc0 0x7ff6680a6180 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.292+0000 7ff68b800640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff68410ab40 0x7ff6841a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.292+0000 7ff68b800640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff68410a180 0x7ff68419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.292+0000 7ff68b800640 1 --2- 172.21.15.84:0/3666067459 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff684108f90 0x7ff68419f1d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.292+0000 7ff68b800640 1 -- 172.21.15.84:0/3666067459 >> 172.21.15.84:0/3666067459 conn(0x7ff684077f40 msgr2=0x7ff68410d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:21.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.292+0000 7ff68b800640 1 -- 172.21.15.84:0/3666067459 shutdown_connections 2024-04-28T12:51:21.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.292+0000 7ff68b800640 1 -- 172.21.15.84:0/3666067459 wait complete. 2024-04-28T12:51:21.308 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:43:15.608080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T12:48:32.037809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T12:48:33.240166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T12:48:37.567238+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T12:51:21.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:51:21.512 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.516+0000 7ff21b200640 1 -- 172.21.15.84:0/1218580738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff21c1028c0 msgr2=0x7ff21c102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.513 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.516+0000 7ff21b200640 1 --2- 172.21.15.84:0/1218580738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff21c1028c0 0x7ff21c102ce0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff20c009860 tx=0x7ff20c02a340 comp rx=0 tx=0).stop 2024-04-28T12:51:21.513 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.516+0000 7ff21b200640 1 -- 172.21.15.84:0/1218580738 shutdown_connections 2024-04-28T12:51:21.513 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.516+0000 7ff21b200640 1 --2- 172.21.15.84:0/1218580738 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff21c104470 0x7ff21c10ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.513 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.516+0000 7ff21b200640 1 --2- 172.21.15.84:0/1218580738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c103ab0 0x7ff21c103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.513 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.516+0000 7ff21b200640 1 --2- 172.21.15.84:0/1218580738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff21c1028c0 0x7ff21c102ce0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.513 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.516+0000 7ff21b200640 1 -- 172.21.15.84:0/1218580738 >> 172.21.15.84:0/1218580738 conn(0x7ff21c0fde30 msgr2=0x7ff21c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:21.513 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.516+0000 7ff21b200640 1 -- 172.21.15.84:0/1218580738 shutdown_connections 2024-04-28T12:51:21.513 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.516+0000 7ff21b200640 1 -- 172.21.15.84:0/1218580738 wait complete. 2024-04-28T12:51:21.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.517+0000 7ff21b200640 1 Processor -- start 2024-04-28T12:51:21.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.517+0000 7ff21b200640 1 -- start start 2024-04-28T12:51:21.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.517+0000 7ff21b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff21c1028c0 0x7ff21c19aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:21.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.517+0000 7ff21b200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff21c103ab0 0x7ff21c19b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:21.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff21b200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c104470 0x7ff21c1a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:21.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff21b200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c10e380 con 0x7ff21c104470 2024-04-28T12:51:21.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff21b200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c10e200 con 0x7ff21c1028c0 2024-04-28T12:51:21.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff21b200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c10e500 con 0x7ff21c103ab0 2024-04-28T12:51:21.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff219e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff21c1028c0 0x7ff21c19aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff219e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff21c1028c0 0x7ff21c19aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58658/0 (socket says 172.21.15.84:58658) 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff219e00640 1 -- 172.21.15.84:0/1911980722 learned_addr learned my addr 172.21.15.84:0/1911980722 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff21a800640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c104470 0x7ff21c1a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff211400640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff21c103ab0 0x7ff21c19b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff219e00640 1 -- 172.21.15.84:0/1911980722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff21c103ab0 msgr2=0x7ff21c19b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff219e00640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff21c103ab0 0x7ff21c19b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff219e00640 1 -- 172.21.15.84:0/1911980722 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c104470 msgr2=0x7ff21c1a2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff219e00640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c104470 0x7ff21c1a2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.518+0000 7ff219e00640 1 -- 172.21.15.84:0/1911980722 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21c1a28e0 con 0x7ff21c1028c0 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.519+0000 7ff219e00640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff21c1028c0 0x7ff21c19aca0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0057c0 tx=0x7ff20c032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.519+0000 7ff213400640 1 -- 172.21.15.84:0/1911980722 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c037040 con 0x7ff21c1028c0 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.519+0000 7ff21b200640 1 -- 172.21.15.84:0/1911980722 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff21c1a2b70 con 0x7ff21c1028c0 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.519+0000 7ff213400640 1 -- 172.21.15.84:0/1911980722 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff20c030bd0 con 0x7ff21c1028c0 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.519+0000 7ff213400640 1 -- 172.21.15.84:0/1911980722 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c005280 con 0x7ff21c1028c0 2024-04-28T12:51:21.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.519+0000 7ff21b200640 1 -- 172.21.15.84:0/1911980722 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff21c1a3000 con 0x7ff21c1028c0 2024-04-28T12:51:21.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.520+0000 7ff21b200640 1 -- 172.21.15.84:0/1911980722 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff21c1a31d0 con 0x7ff21c1028c0 2024-04-28T12:51:21.518 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.522+0000 7ff213400640 1 -- 172.21.15.84:0/1911980722 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff20c043020 con 0x7ff21c1028c0 2024-04-28T12:51:21.520 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.524+0000 7ff213400640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff2000a3c40 0x7ff2000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:21.520 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.524+0000 7ff213400640 1 -- 172.21.15.84:0/1911980722 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff20c0f07d0 con 0x7ff21c1028c0 2024-04-28T12:51:21.520 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.524+0000 7ff211400640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff2000a3c40 0x7ff2000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:21.521 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.524+0000 7ff211400640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff2000a3c40 0x7ff2000a6100 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff21c19c1a0 tx=0x7ff204009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:21.522 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.526+0000 7ff213400640 1 -- 172.21.15.84:0/1911980722 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff20c0bcd80 con 0x7ff21c1028c0 2024-04-28T12:51:21.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.731+0000 7ff21b200640 1 -- 172.21.15.84:0/1911980722 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff21c1a3560 con 0x7ff21c1028c0 2024-04-28T12:51:21.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.732+0000 7ff213400640 1 -- 172.21.15.84:0/1911980722 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7ff20c0328c0 con 0x7ff21c1028c0 2024-04-28T12:51:21.728 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-04-28T12:51:21.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.735+0000 7ff21b200640 1 -- 172.21.15.84:0/1911980722 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff2000a3c40 msgr2=0x7ff2000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.735+0000 7ff21b200640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff2000a3c40 0x7ff2000a6100 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff21c19c1a0 tx=0x7ff204009260 comp rx=0 tx=0).stop 2024-04-28T12:51:21.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.735+0000 7ff21b200640 1 -- 172.21.15.84:0/1911980722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff21c1028c0 msgr2=0x7ff21c19aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.735+0000 7ff21b200640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff21c1028c0 0x7ff21c19aca0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0057c0 tx=0x7ff20c032da0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.735+0000 7ff21b200640 1 -- 172.21.15.84:0/1911980722 shutdown_connections 2024-04-28T12:51:21.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.735+0000 7ff21b200640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff2000a3c40 0x7ff2000a6100 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.735+0000 7ff21b200640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c104470 0x7ff21c1a2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.735+0000 7ff21b200640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff21c103ab0 0x7ff21c19b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.735+0000 7ff21b200640 1 --2- 172.21.15.84:0/1911980722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff21c1028c0 0x7ff21c19aca0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.735+0000 7ff21b200640 1 -- 172.21.15.84:0/1911980722 >> 172.21.15.84:0/1911980722 conn(0x7ff21c0fde30 msgr2=0x7ff21c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:21.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.735+0000 7ff21b200640 1 -- 172.21.15.84:0/1911980722 shutdown_connections 2024-04-28T12:51:21.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.735+0000 7ff21b200640 1 -- 172.21.15.84:0/1911980722 wait complete. 2024-04-28T12:51:21.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T12:51:21.952 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.955+0000 7fc8cb400640 1 -- 172.21.15.84:0/3655061233 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8c410ab40 msgr2=0x7fc8c41113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.952 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.955+0000 7fc8cb400640 1 --2- 172.21.15.84:0/3655061233 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8c410ab40 0x7fc8c41113d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009ca0 tx=0x7fc8bc02a8c0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.952 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.955+0000 7fc8cb400640 1 -- 172.21.15.84:0/3655061233 shutdown_connections 2024-04-28T12:51:21.952 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.955+0000 7fc8cb400640 1 --2- 172.21.15.84:0/3655061233 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8c410ab40 0x7fc8c41113d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.952 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.955+0000 7fc8cb400640 1 --2- 172.21.15.84:0/3655061233 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8c410a180 0x7fc8c410a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.952 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.955+0000 7fc8cb400640 1 --2- 172.21.15.84:0/3655061233 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c4108f90 0x7fc8c41093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.952 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.955+0000 7fc8cb400640 1 -- 172.21.15.84:0/3655061233 >> 172.21.15.84:0/3655061233 conn(0x7fc8c4077f30 msgr2=0x7fc8c407a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:21.952 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.955+0000 7fc8cb400640 1 -- 172.21.15.84:0/3655061233 shutdown_connections 2024-04-28T12:51:21.952 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.955+0000 7fc8cb400640 1 -- 172.21.15.84:0/3655061233 wait complete. 2024-04-28T12:51:21.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.956+0000 7fc8cb400640 1 Processor -- start 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.956+0000 7fc8cb400640 1 -- start start 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8cb400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c4108f90 0x7fc8c419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8cb400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8c410a180 0x7fc8c419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8cb400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8c410ab40 0x7fc8c41a6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8cb400640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c4114b20 con 0x7fc8c410ab40 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8cb400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c41149a0 con 0x7fc8c4108f90 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8cb400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c4114ca0 con 0x7fc8c410a180 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8c9400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8c410ab40 0x7fc8c41a6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8c8a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c4108f90 0x7fc8c419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8c3e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8c410a180 0x7fc8c419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8c9400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8c410ab40 0x7fc8c41a6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42952/0 (socket says 172.21.15.84:42952) 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8c9400640 1 -- 172.21.15.84:0/2822444543 learned_addr learned my addr 172.21.15.84:0/2822444543 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8c8a00640 1 -- 172.21.15.84:0/2822444543 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8c410a180 msgr2=0x7fc8c419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8c8a00640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8c410a180 0x7fc8c419f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8c8a00640 1 -- 172.21.15.84:0/2822444543 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8c410ab40 msgr2=0x7fc8c41a6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:21.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8c8a00640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8c410ab40 0x7fc8c41a6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:21.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8c8a00640 1 -- 172.21.15.84:0/2822444543 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c41a6d90 con 0x7fc8c4108f90 2024-04-28T12:51:21.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.957+0000 7fc8c9400640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8c410ab40 0x7fc8c41a6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:21.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.958+0000 7fc8c8a00640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c4108f90 0x7fc8c419f170 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800e9c0 tx=0x7fc8b800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:21.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.958+0000 7fc8c1600640 1 -- 172.21.15.84:0/2822444543 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b800ccc0 con 0x7fc8c4108f90 2024-04-28T12:51:21.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.958+0000 7fc8c1600640 1 -- 172.21.15.84:0/2822444543 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8b80035f0 con 0x7fc8c4108f90 2024-04-28T12:51:21.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.958+0000 7fc8cb400640 1 -- 172.21.15.84:0/2822444543 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8c41a7080 con 0x7fc8c4108f90 2024-04-28T12:51:21.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.958+0000 7fc8c1600640 1 -- 172.21.15.84:0/2822444543 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b800ce60 con 0x7fc8c4108f90 2024-04-28T12:51:21.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.958+0000 7fc8cb400640 1 -- 172.21.15.84:0/2822444543 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8c41a75d0 con 0x7fc8c4108f90 2024-04-28T12:51:21.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.959+0000 7fc8cb400640 1 -- 172.21.15.84:0/2822444543 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8c41102b0 con 0x7fc8c4108f90 2024-04-28T12:51:21.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.961+0000 7fc8c1600640 1 -- 172.21.15.84:0/2822444543 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc8b8004000 con 0x7fc8c4108f90 2024-04-28T12:51:21.959 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.963+0000 7fc8c1600640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc8ac0a3c40 0x7fc8ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:21.959 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.963+0000 7fc8c1600640 1 -- 172.21.15.84:0/2822444543 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fc8b80d0d90 con 0x7fc8c4108f90 2024-04-28T12:51:21.959 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.963+0000 7fc8c3e00640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc8ac0a3c40 0x7fc8ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:21.960 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.964+0000 7fc8c3e00640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc8ac0a3c40 0x7fc8ac0a6100 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc8c41a0680 tx=0x7fc8b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:21.963 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:21.967+0000 7fc8c1600640 1 -- 172.21.15.84:0/2822444543 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc8b809d230 con 0x7fc8c4108f90 2024-04-28T12:51:22.164 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.167+0000 7fc8cb400640 1 -- 172.21.15.84:0/2822444543 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc8c4063530 con 0x7fc8c4108f90 2024-04-28T12:51:22.164 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.168+0000 7fc8c1600640 1 -- 172.21.15.84:0/2822444543 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fc8b809cb30 con 0x7fc8c4108f90 2024-04-28T12:51:22.165 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T12:51:22.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:21 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3666067459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:51:22.165 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:21 smithi179 ceph-mon[63029]: pgmap v410: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:22.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:21 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1911980722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:51:22.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:21 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3666067459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:51:22.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:21 smithi084 ceph-mon[28081]: pgmap v410: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:22.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:21 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1911980722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:51:22.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.171+0000 7fc8cb400640 1 -- 172.21.15.84:0/2822444543 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc8ac0a3c40 msgr2=0x7fc8ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:22.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.171+0000 7fc8cb400640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc8ac0a3c40 0x7fc8ac0a6100 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc8c41a0680 tx=0x7fc8b4005f90 comp rx=0 tx=0).stop 2024-04-28T12:51:22.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.171+0000 7fc8cb400640 1 -- 172.21.15.84:0/2822444543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c4108f90 msgr2=0x7fc8c419f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:22.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.171+0000 7fc8cb400640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c4108f90 0x7fc8c419f170 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800e9c0 tx=0x7fc8b800ee80 comp rx=0 tx=0).stop 2024-04-28T12:51:22.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.171+0000 7fc8cb400640 1 -- 172.21.15.84:0/2822444543 shutdown_connections 2024-04-28T12:51:22.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.171+0000 7fc8cb400640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc8ac0a3c40 0x7fc8ac0a6100 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.171+0000 7fc8cb400640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8c410ab40 0x7fc8c41a6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.169 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.171+0000 7fc8cb400640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8c410a180 0x7fc8c419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.169 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.171+0000 7fc8cb400640 1 --2- 172.21.15.84:0/2822444543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8c4108f90 0x7fc8c419f170 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.169 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.171+0000 7fc8cb400640 1 -- 172.21.15.84:0/2822444543 >> 172.21.15.84:0/2822444543 conn(0x7fc8c4077f30 msgr2=0x7fc8c410d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:22.169 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.171+0000 7fc8cb400640 1 -- 172.21.15.84:0/2822444543 shutdown_connections 2024-04-28T12:51:22.169 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.171+0000 7fc8cb400640 1 -- 172.21.15.84:0/2822444543 wait complete. 2024-04-28T12:51:22.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T12:51:22.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:21 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3666067459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:51:22.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:21 smithi161 ceph-mon[62387]: pgmap v410: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:22.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:21 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1911980722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:51:22.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.394+0000 7faaf3000640 1 -- 172.21.15.84:0/2315170846 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faaec104300 msgr2=0x7faaec10ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:22.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.394+0000 7faaf3000640 1 --2- 172.21.15.84:0/2315170846 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faaec104300 0x7faaec10ab90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7faae4009ca0 tx=0x7faae402a8c0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.394+0000 7faaf3000640 1 -- 172.21.15.84:0/2315170846 shutdown_connections 2024-04-28T12:51:22.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.394+0000 7faaf3000640 1 --2- 172.21.15.84:0/2315170846 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faaec104300 0x7faaec10ab90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.394+0000 7faaf3000640 1 --2- 172.21.15.84:0/2315170846 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faaec103940 0x7faaec103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.394+0000 7faaf3000640 1 --2- 172.21.15.84:0/2315170846 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaec102750 0x7faaec102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.394+0000 7faaf3000640 1 -- 172.21.15.84:0/2315170846 >> 172.21.15.84:0/2315170846 conn(0x7faaec0fdcc0 msgr2=0x7faaec100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:22.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.394+0000 7faaf3000640 1 -- 172.21.15.84:0/2315170846 shutdown_connections 2024-04-28T12:51:22.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.394+0000 7faaf3000640 1 -- 172.21.15.84:0/2315170846 wait complete. 2024-04-28T12:51:22.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.395+0000 7faaf3000640 1 Processor -- start 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.395+0000 7faaf3000640 1 -- start start 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.395+0000 7faaf3000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faaec102750 0x7faaec1988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.395+0000 7faaf3000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faaec103940 0x7faaec198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf3000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaec104300 0x7faaec19fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf3000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7faaec10e280 con 0x7faaec102750 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf3000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faaec10e100 con 0x7faaec104300 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf3000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7faaec10e400 con 0x7faaec103940 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf2600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaec104300 0x7faaec19fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf2600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaec104300 0x7faaec19fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58694/0 (socket says 172.21.15.84:58694) 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf2600640 1 -- 172.21.15.84:0/3734775580 learned_addr learned my addr 172.21.15.84:0/3734775580 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf1c00640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faaec102750 0x7faaec1988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf1200640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faaec103940 0x7faaec198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf2600640 1 -- 172.21.15.84:0/3734775580 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faaec103940 msgr2=0x7faaec198e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:22.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf2600640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faaec103940 0x7faaec198e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf2600640 1 -- 172.21.15.84:0/3734775580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faaec102750 msgr2=0x7faaec1988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:22.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf2600640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faaec102750 0x7faaec1988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.396+0000 7faaf2600640 1 -- 172.21.15.84:0/3734775580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaec1a0510 con 0x7faaec104300 2024-04-28T12:51:22.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.397+0000 7faaf2600640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaec104300 0x7faaec19fea0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faae40097b0 tx=0x7faae4031770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:22.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.397+0000 7faaea000640 1 -- 172.21.15.84:0/3734775580 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae4003b70 con 0x7faaec104300 2024-04-28T12:51:22.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.397+0000 7faaf3000640 1 -- 172.21.15.84:0/3734775580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaec1a07a0 con 0x7faaec104300 2024-04-28T12:51:22.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.397+0000 7faaea000640 1 -- 172.21.15.84:0/3734775580 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faae4003d10 con 0x7faaec104300 2024-04-28T12:51:22.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.397+0000 7faaea000640 1 -- 172.21.15.84:0/3734775580 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae402fc10 con 0x7faaec104300 2024-04-28T12:51:22.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.397+0000 7faaf3000640 1 -- 172.21.15.84:0/3734775580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaec1a0cf0 con 0x7faaec104300 2024-04-28T12:51:22.398 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.398+0000 7faaf3000640 1 -- 172.21.15.84:0/3734775580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaec109b40 con 0x7faaec104300 2024-04-28T12:51:22.399 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.401+0000 7faaea000640 1 -- 172.21.15.84:0/3734775580 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faae4004040 con 0x7faaec104300 2024-04-28T12:51:22.399 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.402+0000 7faaea000640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7faad40a3c40 0x7faad40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:22.399 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.402+0000 7faaea000640 1 -- 172.21.15.84:0/3734775580 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7faae40f1f60 con 0x7faaec104300 2024-04-28T12:51:22.399 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.402+0000 7faaf1c00640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7faad40a3c40 0x7faad40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:22.399 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.403+0000 7faaf1c00640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7faad40a3c40 0x7faad40a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faaec1037a0 tx=0x7faae0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:22.400 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.404+0000 7faaea000640 1 -- 172.21.15.84:0/3734775580 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faae40be510 con 0x7faaec104300 2024-04-28T12:51:22.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.611+0000 7faaf3000640 1 -- 172.21.15.84:0/3734775580 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faaec063530 con 0x7faaec104300 2024-04-28T12:51:22.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.613+0000 7faaea000640 1 -- 172.21.15.84:0/3734775580 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7faae40bde10 con 0x7faaec104300 2024-04-28T12:51:22.609 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 64 2024-04-28T12:51:22.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.615+0000 7faaf3000640 1 -- 172.21.15.84:0/3734775580 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7faad40a3c40 msgr2=0x7faad40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:22.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.615+0000 7faaf3000640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7faad40a3c40 0x7faad40a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faaec1037a0 tx=0x7faae0005f90 comp rx=0 tx=0).stop 2024-04-28T12:51:22.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.615+0000 7faaf3000640 1 -- 172.21.15.84:0/3734775580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaec104300 msgr2=0x7faaec19fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:22.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.615+0000 7faaf3000640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaec104300 0x7faaec19fea0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faae40097b0 tx=0x7faae4031770 comp rx=0 tx=0).stop 2024-04-28T12:51:22.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.615+0000 7faaf3000640 1 -- 172.21.15.84:0/3734775580 shutdown_connections 2024-04-28T12:51:22.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.615+0000 7faaf3000640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7faad40a3c40 0x7faad40a6100 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.615+0000 7faaf3000640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaec104300 0x7faaec19fea0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.615+0000 7faaf3000640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faaec103940 0x7faaec198e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.615+0000 7faaf3000640 1 --2- 172.21.15.84:0/3734775580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faaec102750 0x7faaec1988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.615+0000 7faaf3000640 1 -- 172.21.15.84:0/3734775580 >> 172.21.15.84:0/3734775580 conn(0x7faaec0fdcc0 msgr2=0x7faaec106d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:22.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.615+0000 7faaf3000640 1 -- 172.21.15.84:0/3734775580 shutdown_connections 2024-04-28T12:51:22.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.615+0000 7faaf3000640 1 -- 172.21.15.84:0/3734775580 wait complete. 2024-04-28T12:51:22.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T12:51:22.840 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.843+0000 7f5bb4c00640 1 -- 172.21.15.84:0/3546107681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0103410 msgr2=0x7f5bb0105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:22.840 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.843+0000 7f5bb4c00640 1 --2- 172.21.15.84:0/3546107681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0103410 0x7f5bb0105800 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0009860 tx=0x7f5ba002a340 comp rx=0 tx=0).stop 2024-04-28T12:51:22.840 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.843+0000 7f5bb4c00640 1 -- 172.21.15.84:0/3546107681 shutdown_connections 2024-04-28T12:51:22.840 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.843+0000 7f5bb4c00640 1 --2- 172.21.15.84:0/3546107681 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0108670 0x7f5bb010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.840 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.843+0000 7f5bb4c00640 1 --2- 172.21.15.84:0/3546107681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0105d40 0x7f5bb0108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.840 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.843+0000 7f5bb4c00640 1 --2- 172.21.15.84:0/3546107681 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0103410 0x7f5bb0105800 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.840 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.843+0000 7f5bb4c00640 1 -- 172.21.15.84:0/3546107681 >> 172.21.15.84:0/3546107681 conn(0x7f5bb00fce00 msgr2=0x7f5bb00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:22.840 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.843+0000 7f5bb4c00640 1 -- 172.21.15.84:0/3546107681 shutdown_connections 2024-04-28T12:51:22.841 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.844+0000 7f5bb4c00640 1 -- 172.21.15.84:0/3546107681 wait complete. 2024-04-28T12:51:22.842 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.845+0000 7f5bb4c00640 1 Processor -- start 2024-04-28T12:51:22.842 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.845+0000 7f5bb4c00640 1 -- start start 2024-04-28T12:51:22.842 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.845+0000 7f5bb4c00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0103410 0x7f5bb019ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.845+0000 7f5bb4c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0105d40 0x7f5bb019b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.845+0000 7f5bb4c00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0108670 0x7f5bb01a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.845+0000 7f5bb4c00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb010e4c0 con 0x7f5bb0103410 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.845+0000 7f5bb4c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb010e340 con 0x7f5bb0105d40 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.845+0000 7f5bb4c00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb010e640 con 0x7f5bb0108670 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.845+0000 7f5bade00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0103410 0x7f5bb019ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.846+0000 7f5bade00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0103410 0x7f5bb019ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42974/0 (socket says 172.21.15.84:42974) 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.846+0000 7f5bade00640 1 -- 172.21.15.84:0/1756353477 learned_addr learned my addr 172.21.15.84:0/1756353477 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.846+0000 7f5bae800640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0108670 0x7f5bb01a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.846+0000 7f5bad400640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0105d40 0x7f5bb019b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.846+0000 7f5bade00640 1 -- 172.21.15.84:0/1756353477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0108670 msgr2=0x7f5bb01a23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.846+0000 7f5bade00640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0108670 0x7f5bb01a23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.846+0000 7f5bade00640 1 -- 172.21.15.84:0/1756353477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0105d40 msgr2=0x7f5bb019b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:22.843 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.846+0000 7f5bade00640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0105d40 0x7f5bb019b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:22.844 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.846+0000 7f5bade00640 1 -- 172.21.15.84:0/1756353477 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb01a2a60 con 0x7f5bb0103410 2024-04-28T12:51:22.844 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.846+0000 7f5bade00640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0103410 0x7f5bb019ae40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0005940 tx=0x7f5ba0030e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:22.845 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.847+0000 7f5ba6a00640 1 -- 172.21.15.84:0/1756353477 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0040020 con 0x7f5bb0103410 2024-04-28T12:51:22.845 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.847+0000 7f5bb4c00640 1 -- 172.21.15.84:0/1756353477 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb01a2cf0 con 0x7f5bb0103410 2024-04-28T12:51:22.845 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.847+0000 7f5ba6a00640 1 -- 172.21.15.84:0/1756353477 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ba002ada0 con 0x7f5bb0103410 2024-04-28T12:51:22.845 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.847+0000 7f5ba6a00640 1 -- 172.21.15.84:0/1756353477 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0037040 con 0x7f5bb0103410 2024-04-28T12:51:22.845 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.847+0000 7f5bb4c00640 1 -- 172.21.15.84:0/1756353477 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb01a3180 con 0x7f5bb0103410 2024-04-28T12:51:22.845 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.848+0000 7f5bb4c00640 1 -- 172.21.15.84:0/1756353477 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb010a1c0 con 0x7f5bb0103410 2024-04-28T12:51:22.846 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.850+0000 7f5ba6a00640 1 -- 172.21.15.84:0/1756353477 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5ba0043020 con 0x7f5bb0103410 2024-04-28T12:51:22.848 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.851+0000 7f5ba6a00640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5b940a3c40 0x7f5b940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:22.848 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.851+0000 7f5ba6a00640 1 -- 172.21.15.84:0/1756353477 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f5ba00f06a0 con 0x7f5bb0103410 2024-04-28T12:51:22.848 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.852+0000 7f5bad400640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5b940a3c40 0x7f5b940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:22.848 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.852+0000 7f5bad400640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5b940a3c40 0x7f5b940a6100 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5bb019c350 tx=0x7f5b9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:22.852 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:22.856+0000 7f5ba6a00640 1 -- 172.21.15.84:0/1756353477 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5ba004a020 con 0x7f5bb0103410 2024-04-28T12:51:22.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:22 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2822444543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:51:22.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:22 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3734775580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:51:23.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.055+0000 7f5bb4c00640 1 -- 172.21.15.84:0/1756353477 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5bb00635d0 con 0x7f5bb0103410 2024-04-28T12:51:23.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.056+0000 7f5ba6a00640 1 -- 172.21.15.84:0/1756353477 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f5ba00bcc50 con 0x7f5bb0103410 2024-04-28T12:51:23.053 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T12:51:23.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.059+0000 7f5bb4c00640 1 -- 172.21.15.84:0/1756353477 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5b940a3c40 msgr2=0x7f5b940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:23.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.059+0000 7f5bb4c00640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5b940a3c40 0x7f5b940a6100 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5bb019c350 tx=0x7f5b9c009260 comp rx=0 tx=0).stop 2024-04-28T12:51:23.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.059+0000 7f5bb4c00640 1 -- 172.21.15.84:0/1756353477 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0103410 msgr2=0x7f5bb019ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:23.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.059+0000 7f5bb4c00640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0103410 0x7f5bb019ae40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0005940 tx=0x7f5ba0030e00 comp rx=0 tx=0).stop 2024-04-28T12:51:23.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.059+0000 7f5bb4c00640 1 -- 172.21.15.84:0/1756353477 shutdown_connections 2024-04-28T12:51:23.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.059+0000 7f5bb4c00640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5b940a3c40 0x7f5b940a6100 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.059+0000 7f5bb4c00640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5bb0108670 0x7f5bb01a23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.059+0000 7f5bb4c00640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bb0105d40 0x7f5bb019b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.059+0000 7f5bb4c00640 1 --2- 172.21.15.84:0/1756353477 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bb0103410 0x7f5bb019ae40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.059+0000 7f5bb4c00640 1 -- 172.21.15.84:0/1756353477 >> 172.21.15.84:0/1756353477 conn(0x7f5bb00fce00 msgr2=0x7f5bb0102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:23.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.059+0000 7f5bb4c00640 1 -- 172.21.15.84:0/1756353477 shutdown_connections 2024-04-28T12:51:23.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.059+0000 7f5bb4c00640 1 -- 172.21.15.84:0/1756353477 wait complete. 2024-04-28T12:51:23.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:51:23.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:22 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2822444543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:51:23.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:22 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3734775580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:51:23.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:22 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2822444543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:51:23.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:22 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3734775580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:51:23.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.284+0000 7f8443200640 1 -- 172.21.15.84:0/653836885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8444106540 msgr2=0x7f844410cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:23.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.284+0000 7f8443200640 1 --2- 172.21.15.84:0/653836885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8444106540 0x7f844410cdd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f843c009900 tx=0x7f843c02a3f0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.284+0000 7f8443200640 1 -- 172.21.15.84:0/653836885 shutdown_connections 2024-04-28T12:51:23.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.284+0000 7f8443200640 1 --2- 172.21.15.84:0/653836885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8444106540 0x7f844410cdd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.284+0000 7f8443200640 1 --2- 172.21.15.84:0/653836885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8444105b80 0x7f8444106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.284+0000 7f8443200640 1 --2- 172.21.15.84:0/653836885 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8444104990 0x7f8444104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.284+0000 7f8443200640 1 -- 172.21.15.84:0/653836885 >> 172.21.15.84:0/653836885 conn(0x7f84440fff00 msgr2=0x7f8444102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:23.281 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.284+0000 7f8443200640 1 -- 172.21.15.84:0/653836885 shutdown_connections 2024-04-28T12:51:23.281 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.284+0000 7f8443200640 1 -- 172.21.15.84:0/653836885 wait complete. 2024-04-28T12:51:23.281 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.285+0000 7f8443200640 1 Processor -- start 2024-04-28T12:51:23.281 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.285+0000 7f8443200640 1 -- start start 2024-04-28T12:51:23.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.285+0000 7f8443200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8444104990 0x7f844419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:23.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.285+0000 7f8443200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8444105b80 0x7f844419b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:23.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.285+0000 7f8443200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8444106540 0x7f84441a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:23.283 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.285+0000 7f8443200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f8444110510 con 0x7f8444104990 2024-04-28T12:51:23.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.285+0000 7f8443200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8444110390 con 0x7f8444106540 2024-04-28T12:51:23.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.285+0000 7f8443200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f8444110690 con 0x7f8444105b80 2024-04-28T12:51:23.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.286+0000 7f8441e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8444104990 0x7f844419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:23.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.286+0000 7f8441e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8444104990 0x7f844419ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42998/0 (socket says 172.21.15.84:42998) 2024-04-28T12:51:23.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.286+0000 7f8441e00640 1 -- 172.21.15.84:0/2737301810 learned_addr learned my addr 172.21.15.84:0/2737301810 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:23.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.286+0000 7f8442800640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8444106540 0x7f84441a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:23.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.286+0000 7f8441400640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8444105b80 0x7f844419b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:23.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.286+0000 7f8442800640 1 -- 172.21.15.84:0/2737301810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8444105b80 msgr2=0x7f844419b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:23.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.286+0000 7f8442800640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8444105b80 0x7f844419b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.286+0000 7f8442800640 1 -- 172.21.15.84:0/2737301810 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8444104990 msgr2=0x7f844419ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:23.284 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.286+0000 7f8442800640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8444104990 0x7f844419ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.286+0000 7f8442800640 1 -- 172.21.15.84:0/2737301810 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84441a2740 con 0x7f8444106540 2024-04-28T12:51:23.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.286+0000 7f8441e00640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8444104990 0x7f844419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:23.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.286+0000 7f8441400640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8444105b80 0x7f844419b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:51:23.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.287+0000 7f8442800640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8444106540 0x7f84441a20d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f843c02a910 tx=0x7f843c003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:23.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.287+0000 7f843aa00640 1 -- 172.21.15.84:0/2737301810 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c03f070 con 0x7f8444106540 2024-04-28T12:51:23.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.287+0000 7f843aa00640 1 -- 172.21.15.84:0/2737301810 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f843c031c50 con 0x7f8444106540 2024-04-28T12:51:23.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.287+0000 7f8443200640 1 -- 172.21.15.84:0/2737301810 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84441a29d0 con 0x7f8444106540 2024-04-28T12:51:23.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.287+0000 7f843aa00640 1 -- 172.21.15.84:0/2737301810 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c035030 con 0x7f8444106540 2024-04-28T12:51:23.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.287+0000 7f8443200640 1 -- 172.21.15.84:0/2737301810 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84441a2ee0 con 0x7f8444106540 2024-04-28T12:51:23.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.288+0000 7f8443200640 1 -- 172.21.15.84:0/2737301810 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f844410bcb0 con 0x7f8444106540 2024-04-28T12:51:23.286 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.290+0000 7f843aa00640 1 -- 172.21.15.84:0/2737301810 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f843c042070 con 0x7f8444106540 2024-04-28T12:51:23.288 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.292+0000 7f843aa00640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f84240a3cc0 0x7f84240a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:23.288 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.292+0000 7f843aa00640 1 -- 172.21.15.84:0/2737301810 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f843c0f1340 con 0x7f8444106540 2024-04-28T12:51:23.288 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.292+0000 7f8441e00640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f84240a3cc0 0x7f84240a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:23.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.293+0000 7f8441e00640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f84240a3cc0 0x7f84240a6180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f84340058c0 tx=0x7f8434005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:23.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.295+0000 7f843aa00640 1 -- 172.21.15.84:0/2737301810 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f843c0bd7e0 con 0x7f8444106540 2024-04-28T12:51:23.537 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.540+0000 7f8443200640 1 -- 172.21.15.84:0/2737301810 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f844402dd20 con 0x7f8444106540 2024-04-28T12:51:23.538 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.542+0000 7f843aa00640 1 -- 172.21.15.84:0/2737301810 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11884 (secure 0 0 0) 0x7f843c0bd0e0 con 0x7f8444106540 2024-04-28T12:51:23.538 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:51:23.539 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6841/1196924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":1196924806},{"type":"v1","addr":"172.21.15.84:6841","nonce":1196924806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:34.334879+0000","modified":"2024-04-28T12:49:08.738419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24562,"mds_2":24545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6839/3812720642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":3812720642},{"type":"v1","addr":"172.21.15.161:6839","nonce":3812720642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24545,24562,24602]},"id":1}]} 2024-04-28T12:51:23.539 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 24 2024-04-28T12:51:23.541 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.545+0000 7f8443200640 1 -- 172.21.15.84:0/2737301810 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f84240a3cc0 msgr2=0x7f84240a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:23.541 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.545+0000 7f8443200640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f84240a3cc0 0x7f84240a6180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f84340058c0 tx=0x7f8434005f90 comp rx=0 tx=0).stop 2024-04-28T12:51:23.541 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.545+0000 7f8443200640 1 -- 172.21.15.84:0/2737301810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8444106540 msgr2=0x7f84441a20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:23.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.545+0000 7f8443200640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8444106540 0x7f84441a20d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f843c02a910 tx=0x7f843c003b90 comp rx=0 tx=0).stop 2024-04-28T12:51:23.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.545+0000 7f8443200640 1 -- 172.21.15.84:0/2737301810 shutdown_connections 2024-04-28T12:51:23.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.545+0000 7f8443200640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f84240a3cc0 0x7f84240a6180 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.545+0000 7f8443200640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8444106540 0x7f84441a20d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.545+0000 7f8443200640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8444105b80 0x7f844419b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.545+0000 7f8443200640 1 --2- 172.21.15.84:0/2737301810 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8444104990 0x7f844419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.545+0000 7f8443200640 1 -- 172.21.15.84:0/2737301810 >> 172.21.15.84:0/2737301810 conn(0x7f84440fff00 msgr2=0x7f8444108e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:23.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.545+0000 7f8443200640 1 -- 172.21.15.84:0/2737301810 shutdown_connections 2024-04-28T12:51:23.542 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.545+0000 7f8443200640 1 -- 172.21.15.84:0/2737301810 wait complete. 2024-04-28T12:51:23.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:51:23.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.767+0000 7f239cc00640 1 -- 172.21.15.84:0/1006908624 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23980772f0 msgr2=0x7f2398077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:23.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.767+0000 7f239cc00640 1 --2- 172.21.15.84:0/1006908624 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23980772f0 0x7f2398077710 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2388009900 tx=0x7f238802a3f0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.768+0000 7f239cc00640 1 -- 172.21.15.84:0/1006908624 shutdown_connections 2024-04-28T12:51:23.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.768+0000 7f239cc00640 1 --2- 172.21.15.84:0/1006908624 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2398078ed0 0x7f2398079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.768+0000 7f239cc00640 1 --2- 172.21.15.84:0/1006908624 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2398078530 0x7f2398078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.768+0000 7f239cc00640 1 --2- 172.21.15.84:0/1006908624 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23980772f0 0x7f2398077710 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.768+0000 7f239cc00640 1 -- 172.21.15.84:0/1006908624 >> 172.21.15.84:0/1006908624 conn(0x7f23980fff40 msgr2=0x7f2398102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:23.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.768+0000 7f239cc00640 1 -- 172.21.15.84:0/1006908624 shutdown_connections 2024-04-28T12:51:23.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.768+0000 7f239cc00640 1 -- 172.21.15.84:0/1006908624 wait complete. 2024-04-28T12:51:23.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.769+0000 7f239cc00640 1 Processor -- start 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.769+0000 7f239cc00640 1 -- start start 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.769+0000 7f239cc00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23980772f0 0x7f239819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.769+0000 7f239cc00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2398078530 0x7f239819f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.769+0000 7f239cc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2398078ed0 0x7f23981a64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.769+0000 7f239cc00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f23981148f0 con 0x7f23980772f0 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.769+0000 7f239cc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2398114770 con 0x7f2398078ed0 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.769+0000 7f239cc00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f2398114a70 con 0x7f2398078530 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.769+0000 7f2397400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23980772f0 0x7f239819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.770+0000 7f238ea00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2398078530 0x7f239819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.770+0000 7f2397400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23980772f0 0x7f239819ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43014/0 (socket says 172.21.15.84:43014) 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.770+0000 7f2397400640 1 -- 172.21.15.84:0/1460295452 learned_addr learned my addr 172.21.15.84:0/1460295452 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.770+0000 7f2397e00640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2398078ed0 0x7f23981a64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.770+0000 7f2397400640 1 -- 172.21.15.84:0/1460295452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2398078530 msgr2=0x7f239819f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:23.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.770+0000 7f2397400640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2398078530 0x7f239819f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.770+0000 7f2397400640 1 -- 172.21.15.84:0/1460295452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2398078ed0 msgr2=0x7f23981a64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:23.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.770+0000 7f2397400640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2398078ed0 0x7f23981a64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.770+0000 7f2397400640 1 -- 172.21.15.84:0/1460295452 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23981a6b60 con 0x7f23980772f0 2024-04-28T12:51:23.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.770+0000 7f238ea00640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2398078530 0x7f239819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:23.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.770+0000 7f2397400640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23980772f0 0x7f239819ef40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2388005ab0 tx=0x7f2388003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:23.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.771+0000 7f2394c00640 1 -- 172.21.15.84:0/1460295452 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238803f070 con 0x7f23980772f0 2024-04-28T12:51:23.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.771+0000 7f2394c00640 1 -- 172.21.15.84:0/1460295452 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2388031c50 con 0x7f23980772f0 2024-04-28T12:51:23.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.771+0000 7f2394c00640 1 -- 172.21.15.84:0/1460295452 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388035030 con 0x7f23980772f0 2024-04-28T12:51:23.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.771+0000 7f239cc00640 1 -- 172.21.15.84:0/1460295452 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23981a6df0 con 0x7f23980772f0 2024-04-28T12:51:23.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.771+0000 7f239cc00640 1 -- 172.21.15.84:0/1460295452 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23981a7250 con 0x7f23980772f0 2024-04-28T12:51:23.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.772+0000 7f239cc00640 1 -- 172.21.15.84:0/1460295452 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2398111320 con 0x7f23980772f0 2024-04-28T12:51:23.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.774+0000 7f2394c00640 1 -- 172.21.15.84:0/1460295452 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2388042070 con 0x7f23980772f0 2024-04-28T12:51:23.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.775+0000 7f2394c00640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f237c0a3cc0 0x7f237c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:23.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.776+0000 7f2394c00640 1 -- 172.21.15.84:0/1460295452 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f23880f0fa0 con 0x7f23980772f0 2024-04-28T12:51:23.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.776+0000 7f238ea00640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f237c0a3cc0 0x7f237c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:23.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.776+0000 7f238ea00640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f237c0a3cc0 0x7f237c0a6180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f23981a0450 tx=0x7f2380005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:23.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.780+0000 7f2394c00640 1 -- 172.21.15.84:0/1460295452 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f23880bcdf0 con 0x7f23980772f0 2024-04-28T12:51:23.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:23 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1756353477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:51:23.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:23 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2737301810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:51:23.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:23 smithi084 ceph-mon[28081]: pgmap v411: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:23.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.985+0000 7f239cc00640 1 -- 172.21.15.84:0/1460295452 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f239802dcf0 con 0x7f23980772f0 2024-04-28T12:51:23.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.989+0000 7f2394c00640 1 -- 172.21.15.84:0/1460295452 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24634 (secure 0 0 0) 0x7f23880c10f0 con 0x7f23980772f0 2024-04-28T12:51:23.986 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:51:23.987 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":94,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:50:30.848698+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.8","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:51:23.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.992+0000 7f239cc00640 1 -- 172.21.15.84:0/1460295452 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f237c0a3cc0 msgr2=0x7f237c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:23.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.992+0000 7f239cc00640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f237c0a3cc0 0x7f237c0a6180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f23981a0450 tx=0x7f2380005f90 comp rx=0 tx=0).stop 2024-04-28T12:51:23.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.992+0000 7f239cc00640 1 -- 172.21.15.84:0/1460295452 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23980772f0 msgr2=0x7f239819ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:23.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.992+0000 7f239cc00640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23980772f0 0x7f239819ef40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2388005ab0 tx=0x7f2388003b90 comp rx=0 tx=0).stop 2024-04-28T12:51:23.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.993+0000 7f239cc00640 1 -- 172.21.15.84:0/1460295452 shutdown_connections 2024-04-28T12:51:23.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.993+0000 7f239cc00640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f237c0a3cc0 0x7f237c0a6180 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.993+0000 7f239cc00640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2398078ed0 0x7f23981a64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.993+0000 7f239cc00640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2398078530 0x7f239819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.993+0000 7f239cc00640 1 --2- 172.21.15.84:0/1460295452 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23980772f0 0x7f239819ef40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:23.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.993+0000 7f239cc00640 1 -- 172.21.15.84:0/1460295452 >> 172.21.15.84:0/1460295452 conn(0x7f23980fff40 msgr2=0x7f2398101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:23.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.993+0000 7f239cc00640 1 -- 172.21.15.84:0/1460295452 shutdown_connections 2024-04-28T12:51:23.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:23.993+0000 7f239cc00640 1 -- 172.21.15.84:0/1460295452 wait complete. 2024-04-28T12:51:24.008 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:43:15.608080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T12:48:32.037809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T12:48:33.240166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T12:48:37.567238+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T12:51:24.009 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T12:51:24.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:23 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1756353477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:51:24.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:23 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2737301810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:51:24.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:23 smithi179 ceph-mon[63029]: pgmap v411: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:24.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.220+0000 7f1df4c00640 1 -- 172.21.15.84:0/2187028298 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1df0101a70 msgr2=0x7f1df0101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:24.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.220+0000 7f1df4c00640 1 --2- 172.21.15.84:0/2187028298 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1df0101a70 0x7f1df0101ed0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009900 tx=0x7f1dd802a410 comp rx=0 tx=0).stop 2024-04-28T12:51:24.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.220+0000 7f1df4c00640 1 -- 172.21.15.84:0/2187028298 shutdown_connections 2024-04-28T12:51:24.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.220+0000 7f1df4c00640 1 --2- 172.21.15.84:0/2187028298 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1df0102410 0x7f1df0102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.220+0000 7f1df4c00640 1 --2- 172.21.15.84:0/2187028298 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1df0101a70 0x7f1df0101ed0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.220+0000 7f1df4c00640 1 --2- 172.21.15.84:0/2187028298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1df0101110 0x7f1df0101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.220+0000 7f1df4c00640 1 -- 172.21.15.84:0/2187028298 >> 172.21.15.84:0/2187028298 conn(0x7f1df00fccd0 msgr2=0x7f1df00ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:24.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.220+0000 7f1df4c00640 1 -- 172.21.15.84:0/2187028298 shutdown_connections 2024-04-28T12:51:24.218 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.220+0000 7f1df4c00640 1 -- 172.21.15.84:0/2187028298 wait complete. 2024-04-28T12:51:24.220 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.221+0000 7f1df4c00640 1 Processor -- start 2024-04-28T12:51:24.220 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.221+0000 7f1df4c00640 1 -- start start 2024-04-28T12:51:24.220 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.222+0000 7f1df4c00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1df0101110 0x7f1df019ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:24.220 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.222+0000 7f1df4c00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1df0101a70 0x7f1df019b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:24.220 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.222+0000 7f1df4c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1df0102410 0x7f1df01a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:24.220 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.222+0000 7f1df4c00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1df01105b0 con 0x7f1df0101110 2024-04-28T12:51:24.220 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.222+0000 7f1df4c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1df0110430 con 0x7f1df0102410 2024-04-28T12:51:24.220 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.222+0000 7f1df4c00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1df0110730 con 0x7f1df0101a70 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.222+0000 7f1def400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1df0101110 0x7f1df019ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.222+0000 7f1de6a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1df0101a70 0x7f1df019b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.222+0000 7f1def400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1df0101110 0x7f1df019ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43028/0 (socket says 172.21.15.84:43028) 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.222+0000 7f1def400640 1 -- 172.21.15.84:0/3206470492 learned_addr learned my addr 172.21.15.84:0/3206470492 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.222+0000 7f1defe00640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1df0102410 0x7f1df01a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.223+0000 7f1de6a00640 1 -- 172.21.15.84:0/3206470492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1df0102410 msgr2=0x7f1df01a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.223+0000 7f1de6a00640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1df0102410 0x7f1df01a21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.223+0000 7f1de6a00640 1 -- 172.21.15.84:0/3206470492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1df0101110 msgr2=0x7f1df019ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.223+0000 7f1de6a00640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1df0101110 0x7f1df019ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.223+0000 7f1de6a00640 1 -- 172.21.15.84:0/3206470492 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df01a2880 con 0x7f1df0101a70 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.223+0000 7f1def400640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1df0101110 0x7f1df019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.223+0000 7f1de6a00640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1df0101a70 0x7f1df019b1b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1dd802aa00 tx=0x7f1dd8003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.223+0000 7f1decc00640 1 -- 172.21.15.84:0/3206470492 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd803f070 con 0x7f1df0101a70 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.223+0000 7f1df4c00640 1 -- 172.21.15.84:0/3206470492 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df01a2b10 con 0x7f1df0101a70 2024-04-28T12:51:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.223+0000 7f1decc00640 1 -- 172.21.15.84:0/3206470492 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dd8031c50 con 0x7f1df0101a70 2024-04-28T12:51:24.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.223+0000 7f1decc00640 1 -- 172.21.15.84:0/3206470492 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8035030 con 0x7f1df0101a70 2024-04-28T12:51:24.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.223+0000 7f1df4c00640 1 -- 172.21.15.84:0/3206470492 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df01a3020 con 0x7f1df0101a70 2024-04-28T12:51:24.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.224+0000 7f1df4c00640 1 -- 172.21.15.84:0/3206470492 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df010c040 con 0x7f1df0101a70 2024-04-28T12:51:24.223 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.227+0000 7f1decc00640 1 -- 172.21.15.84:0/3206470492 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1dd8042070 con 0x7f1df0101a70 2024-04-28T12:51:24.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.228+0000 7f1decc00640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f1dd40a3cc0 0x7f1dd40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:24.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.228+0000 7f1decc00640 1 -- 172.21.15.84:0/3206470492 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f1dd80f16d0 con 0x7f1df0101a70 2024-04-28T12:51:24.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.228+0000 7f1def400640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f1dd40a3cc0 0x7f1dd40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:24.225 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.229+0000 7f1def400640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f1dd40a3cc0 0x7f1dd40a6180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1df019c140 tx=0x7f1de0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:24.227 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.231+0000 7f1decc00640 1 -- 172.21.15.84:0/3206470492 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1dd80bd520 con 0x7f1df0101a70 2024-04-28T12:51:24.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:23 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1756353477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:51:24.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:23 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2737301810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:51:24.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:23 smithi161 ceph-mon[62387]: pgmap v411: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:24.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.431+0000 7f1df4c00640 1 -- 172.21.15.84:0/3206470492 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1df002dd40 con 0x7f1df0101a70 2024-04-28T12:51:24.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.432+0000 7f1decc00640 1 -- 172.21.15.84:0/3206470492 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f1dd80c1820 con 0x7f1df0101a70 2024-04-28T12:51:24.429 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-04-28T12:51:24.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.435+0000 7f1df4c00640 1 -- 172.21.15.84:0/3206470492 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f1dd40a3cc0 msgr2=0x7f1dd40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:24.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.435+0000 7f1df4c00640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f1dd40a3cc0 0x7f1dd40a6180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1df019c140 tx=0x7f1de0005f90 comp rx=0 tx=0).stop 2024-04-28T12:51:24.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.435+0000 7f1df4c00640 1 -- 172.21.15.84:0/3206470492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1df0101a70 msgr2=0x7f1df019b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:24.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.435+0000 7f1df4c00640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1df0101a70 0x7f1df019b1b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1dd802aa00 tx=0x7f1dd8003c10 comp rx=0 tx=0).stop 2024-04-28T12:51:24.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.435+0000 7f1df4c00640 1 -- 172.21.15.84:0/3206470492 shutdown_connections 2024-04-28T12:51:24.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.435+0000 7f1df4c00640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f1dd40a3cc0 0x7f1dd40a6180 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.435+0000 7f1df4c00640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1df0102410 0x7f1df01a21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.435+0000 7f1df4c00640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1df0101a70 0x7f1df019b1b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.435+0000 7f1df4c00640 1 --2- 172.21.15.84:0/3206470492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1df0101110 0x7f1df019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.435+0000 7f1df4c00640 1 -- 172.21.15.84:0/3206470492 >> 172.21.15.84:0/3206470492 conn(0x7f1df00fccd0 msgr2=0x7f1df00fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:24.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.435+0000 7f1df4c00640 1 -- 172.21.15.84:0/3206470492 shutdown_connections 2024-04-28T12:51:24.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.435+0000 7f1df4c00640 1 -- 172.21.15.84:0/3206470492 wait complete. 2024-04-28T12:51:24.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T12:51:24.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.657+0000 7feade400640 1 -- 172.21.15.84:0/3232695444 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead8109fc0 msgr2=0x7fead810a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:24.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.657+0000 7feade400640 1 --2- 172.21.15.84:0/3232695444 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead8109fc0 0x7fead810a440 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7feac4009900 tx=0x7feac402a3f0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.657+0000 7feade400640 1 -- 172.21.15.84:0/3232695444 shutdown_connections 2024-04-28T12:51:24.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.657+0000 7feade400640 1 --2- 172.21.15.84:0/3232695444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fead810a980 0x7fead81112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.657+0000 7feade400640 1 --2- 172.21.15.84:0/3232695444 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead8109fc0 0x7fead810a440 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.657+0000 7feade400640 1 --2- 172.21.15.84:0/3232695444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fead8108e70 0x7fead8109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.657+0000 7feade400640 1 -- 172.21.15.84:0/3232695444 >> 172.21.15.84:0/3232695444 conn(0x7fead8077e00 msgr2=0x7fead807a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:24.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.657+0000 7feade400640 1 -- 172.21.15.84:0/3232695444 shutdown_connections 2024-04-28T12:51:24.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.657+0000 7feade400640 1 -- 172.21.15.84:0/3232695444 wait complete. 2024-04-28T12:51:24.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.658+0000 7feade400640 1 Processor -- start 2024-04-28T12:51:24.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.658+0000 7feade400640 1 -- start start 2024-04-28T12:51:24.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feade400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fead8108e70 0x7fead819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:24.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feade400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fead8109fc0 0x7fead819f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:24.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feade400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead810a980 0x7fead81a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:24.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feade400640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fead81149c0 con 0x7fead810a980 2024-04-28T12:51:24.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feade400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fead8114840 con 0x7fead8109fc0 2024-04-28T12:51:24.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feade400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fead8114b40 con 0x7fead8108e70 2024-04-28T12:51:24.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feadda00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead810a980 0x7fead81a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:24.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feadda00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead810a980 0x7fead81a6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43048/0 (socket says 172.21.15.84:43048) 2024-04-28T12:51:24.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feadda00640 1 -- 172.21.15.84:0/2695564487 learned_addr learned my addr 172.21.15.84:0/2695564487 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:24.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feadd000640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fead8108e70 0x7fead819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7fead7e00640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fead8109fc0 0x7fead819f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feadda00640 1 -- 172.21.15.84:0/2695564487 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fead8108e70 msgr2=0x7fead819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feadda00640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fead8108e70 0x7fead819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feadda00640 1 -- 172.21.15.84:0/2695564487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fead8109fc0 msgr2=0x7fead819f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feadda00640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fead8109fc0 0x7fead819f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7feadda00640 1 -- 172.21.15.84:0/2695564487 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead81a6cb0 con 0x7fead810a980 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.659+0000 7fead7e00640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fead8109fc0 0x7fead819f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.660+0000 7feadda00640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead810a980 0x7fead81a6640 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fead000eb10 tx=0x7fead000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.660+0000 7fead5600640 1 -- 172.21.15.84:0/2695564487 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0004b20 con 0x7fead810a980 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.660+0000 7fead5600640 1 -- 172.21.15.84:0/2695564487 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fead0004cc0 con 0x7fead810a980 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.660+0000 7fead5600640 1 -- 172.21.15.84:0/2695564487 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0010670 con 0x7fead810a980 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.660+0000 7feade400640 1 -- 172.21.15.84:0/2695564487 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead81a6fa0 con 0x7fead810a980 2024-04-28T12:51:24.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.660+0000 7feade400640 1 -- 172.21.15.84:0/2695564487 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead81a7410 con 0x7fead810a980 2024-04-28T12:51:24.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.661+0000 7feade400640 1 -- 172.21.15.84:0/2695564487 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fead8110200 con 0x7fead810a980 2024-04-28T12:51:24.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.663+0000 7fead5600640 1 -- 172.21.15.84:0/2695564487 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fead0005020 con 0x7fead810a980 2024-04-28T12:51:24.662 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.665+0000 7fead5600640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7feac00a3cc0 0x7feac00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:24.662 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.666+0000 7fead5600640 1 -- 172.21.15.84:0/2695564487 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fead00d0f00 con 0x7fead810a980 2024-04-28T12:51:24.662 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.666+0000 7feadd000640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7feac00a3cc0 0x7feac00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:24.663 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.667+0000 7feadd000640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7feac00a3cc0 0x7feac00a6180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7feacc009bd0 tx=0x7feacc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:24.667 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.670+0000 7fead5600640 1 -- 172.21.15.84:0/2695564487 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fead009d3a0 con 0x7fead810a980 2024-04-28T12:51:24.870 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.873+0000 7feade400640 1 -- 172.21.15.84:0/2695564487 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fead802dd20 con 0x7fead810a980 2024-04-28T12:51:24.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.895+0000 7fead5600640 1 -- 172.21.15.84:0/2695564487 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fead009cca0 con 0x7fead810a980 2024-04-28T12:51:24.892 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T12:51:24.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.899+0000 7feade400640 1 -- 172.21.15.84:0/2695564487 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7feac00a3cc0 msgr2=0x7feac00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:24.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.899+0000 7feade400640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7feac00a3cc0 0x7feac00a6180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7feacc009bd0 tx=0x7feacc005f90 comp rx=0 tx=0).stop 2024-04-28T12:51:24.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.899+0000 7feade400640 1 -- 172.21.15.84:0/2695564487 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead810a980 msgr2=0x7fead81a6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:24.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.899+0000 7feade400640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead810a980 0x7fead81a6640 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fead000eb10 tx=0x7fead000c2d0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.899+0000 7feade400640 1 -- 172.21.15.84:0/2695564487 shutdown_connections 2024-04-28T12:51:24.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.899+0000 7feade400640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7feac00a3cc0 0x7feac00a6180 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.899+0000 7feade400640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead810a980 0x7fead81a6640 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.899+0000 7feade400640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fead8109fc0 0x7fead819f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.899+0000 7feade400640 1 --2- 172.21.15.84:0/2695564487 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fead8108e70 0x7fead819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:24.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.899+0000 7feade400640 1 -- 172.21.15.84:0/2695564487 >> 172.21.15.84:0/2695564487 conn(0x7fead8077e00 msgr2=0x7fead810d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:24.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.899+0000 7feade400640 1 -- 172.21.15.84:0/2695564487 shutdown_connections 2024-04-28T12:51:24.896 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:24.899+0000 7feade400640 1 -- 172.21.15.84:0/2695564487 wait complete. 2024-04-28T12:51:24.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T12:51:25.114 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.117+0000 7fb7bf200640 1 -- 172.21.15.84:0/1697838411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7c0103930 msgr2=0x7fb7c0105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.114 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.117+0000 7fb7bf200640 1 --2- 172.21.15.84:0/1697838411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7c0103930 0x7fb7c0105da0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009900 tx=0x7fb7a802a3f0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.114 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.117+0000 7fb7bf200640 1 -- 172.21.15.84:0/1697838411 shutdown_connections 2024-04-28T12:51:25.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.117+0000 7fb7bf200640 1 --2- 172.21.15.84:0/1697838411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c01062e0 0x7fb7c0106760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.117+0000 7fb7bf200640 1 --2- 172.21.15.84:0/1697838411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7c0103930 0x7fb7c0105da0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.117+0000 7fb7bf200640 1 --2- 172.21.15.84:0/1697838411 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7c0100fc0 0x7fb7c01033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.117+0000 7fb7bf200640 1 -- 172.21.15.84:0/1697838411 >> 172.21.15.84:0/1697838411 conn(0x7fb7c00fcbc0 msgr2=0x7fb7c00ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:25.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.118+0000 7fb7bf200640 1 -- 172.21.15.84:0/1697838411 shutdown_connections 2024-04-28T12:51:25.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.118+0000 7fb7bf200640 1 -- 172.21.15.84:0/1697838411 wait complete. 2024-04-28T12:51:25.117 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.118+0000 7fb7bf200640 1 Processor -- start 2024-04-28T12:51:25.117 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.119+0000 7fb7bf200640 1 -- start start 2024-04-28T12:51:25.117 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.119+0000 7fb7bf200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7c0100fc0 0x7fb7c019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:25.117 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.119+0000 7fb7bf200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7c0103930 0x7fb7c019f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:25.117 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.119+0000 7fb7bf200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c01062e0 0x7fb7c01a64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:25.117 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.119+0000 7fb7bf200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c01148a0 con 0x7fb7c0100fc0 2024-04-28T12:51:25.117 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.119+0000 7fb7bf200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c0114720 con 0x7fb7c01062e0 2024-04-28T12:51:25.117 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.119+0000 7fb7bf200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c0114a20 con 0x7fb7c0103930 2024-04-28T12:51:25.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.119+0000 7fb7bde00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7c0100fc0 0x7fb7c019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:25.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.119+0000 7fb7bde00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7c0100fc0 0x7fb7c019ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43072/0 (socket says 172.21.15.84:43072) 2024-04-28T12:51:25.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.119+0000 7fb7bde00640 1 -- 172.21.15.84:0/609327 learned_addr learned my addr 172.21.15.84:0/609327 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:25.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.119+0000 7fb7b7e00640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7c0103930 0x7fb7c019f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:25.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.120+0000 7fb7be800640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c01062e0 0x7fb7c01a64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:25.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.120+0000 7fb7bde00640 1 -- 172.21.15.84:0/609327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7c0103930 msgr2=0x7fb7c019f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.120+0000 7fb7bde00640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7c0103930 0x7fb7c019f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.120+0000 7fb7bde00640 1 -- 172.21.15.84:0/609327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c01062e0 msgr2=0x7fb7c01a64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.120+0000 7fb7bde00640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c01062e0 0x7fb7c01a64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.120+0000 7fb7bde00640 1 -- 172.21.15.84:0/609327 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c01a6b30 con 0x7fb7c0100fc0 2024-04-28T12:51:25.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.120+0000 7fb7bde00640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7c0100fc0 0x7fb7c019ef10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb7b000d950 tx=0x7fb7b000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:25.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.120+0000 7fb7b6a00640 1 -- 172.21.15.84:0/609327 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0017030 con 0x7fb7c0100fc0 2024-04-28T12:51:25.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.120+0000 7fb7b6a00640 1 -- 172.21.15.84:0/609327 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7b0004020 con 0x7fb7c0100fc0 2024-04-28T12:51:25.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.120+0000 7fb7b6a00640 1 -- 172.21.15.84:0/609327 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b00047a0 con 0x7fb7c0100fc0 2024-04-28T12:51:25.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.120+0000 7fb7bf200640 1 -- 172.21.15.84:0/609327 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c01a6e20 con 0x7fb7c0100fc0 2024-04-28T12:51:25.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.120+0000 7fb7bf200640 1 -- 172.21.15.84:0/609327 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c01a71d0 con 0x7fb7c0100fc0 2024-04-28T12:51:25.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.121+0000 7fb7bf200640 1 -- 172.21.15.84:0/609327 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7c00fe6b0 con 0x7fb7c0100fc0 2024-04-28T12:51:25.120 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.124+0000 7fb7b6a00640 1 -- 172.21.15.84:0/609327 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb7b0009a30 con 0x7fb7c0100fc0 2024-04-28T12:51:25.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.126+0000 7fb7b6a00640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb7a40a3c40 0x7fb7a40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:25.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.126+0000 7fb7b6a00640 1 -- 172.21.15.84:0/609327 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fb7b00d03d0 con 0x7fb7c0100fc0 2024-04-28T12:51:25.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.126+0000 7fb7b7e00640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb7a40a3c40 0x7fb7a40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:25.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.127+0000 7fb7b7e00640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb7a40a3c40 0x7fb7a40a6100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8031500 tx=0x7fb7a8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:25.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.130+0000 7fb7b6a00640 1 -- 172.21.15.84:0/609327 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb7b009c8d0 con 0x7fb7c0100fc0 2024-04-28T12:51:25.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:24 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1460295452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:51:25.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:24 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3206470492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:51:25.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:24 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1460295452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:51:25.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:24 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3206470492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:51:25.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:24 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1460295452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:51:25.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:24 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3206470492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T12:51:25.326 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.330+0000 7fb7bf200640 1 -- 172.21.15.84:0/609327 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb7c002dd50 con 0x7fb7c0100fc0 2024-04-28T12:51:25.327 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.331+0000 7fb7b6a00640 1 -- 172.21.15.84:0/609327 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fb7b009c1d0 con 0x7fb7c0100fc0 2024-04-28T12:51:25.327 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 64 2024-04-28T12:51:25.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.333+0000 7fb7bf200640 1 -- 172.21.15.84:0/609327 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb7a40a3c40 msgr2=0x7fb7a40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.333+0000 7fb7bf200640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb7a40a3c40 0x7fb7a40a6100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8031500 tx=0x7fb7a8005a20 comp rx=0 tx=0).stop 2024-04-28T12:51:25.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.333+0000 7fb7bf200640 1 -- 172.21.15.84:0/609327 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7c0100fc0 msgr2=0x7fb7c019ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.333+0000 7fb7bf200640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7c0100fc0 0x7fb7c019ef10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb7b000d950 tx=0x7fb7b000de10 comp rx=0 tx=0).stop 2024-04-28T12:51:25.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.334+0000 7fb7bf200640 1 -- 172.21.15.84:0/609327 shutdown_connections 2024-04-28T12:51:25.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.334+0000 7fb7bf200640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fb7a40a3c40 0x7fb7a40a6100 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.334+0000 7fb7bf200640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7c01062e0 0x7fb7c01a64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.334+0000 7fb7bf200640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7c0103930 0x7fb7c019f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.334+0000 7fb7bf200640 1 --2- 172.21.15.84:0/609327 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb7c0100fc0 0x7fb7c019ef10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.334+0000 7fb7bf200640 1 -- 172.21.15.84:0/609327 >> 172.21.15.84:0/609327 conn(0x7fb7c00fcbc0 msgr2=0x7fb7c0101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:25.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.334+0000 7fb7bf200640 1 -- 172.21.15.84:0/609327 shutdown_connections 2024-04-28T12:51:25.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.334+0000 7fb7bf200640 1 -- 172.21.15.84:0/609327 wait complete. 2024-04-28T12:51:25.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T12:51:25.546 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.549+0000 7fc296600640 1 -- 172.21.15.84:0/900311955 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc29010a180 msgr2=0x7fc29010a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.549+0000 7fc296600640 1 --2- 172.21.15.84:0/900311955 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc29010a180 0x7fc29010a600 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc278009900 tx=0x7fc27802a3f0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.549+0000 7fc296600640 1 -- 172.21.15.84:0/900311955 shutdown_connections 2024-04-28T12:51:25.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.549+0000 7fc296600640 1 --2- 172.21.15.84:0/900311955 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc29010ab40 0x7fc2901113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.549+0000 7fc296600640 1 --2- 172.21.15.84:0/900311955 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc29010a180 0x7fc29010a600 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.549+0000 7fc296600640 1 --2- 172.21.15.84:0/900311955 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc290108f90 0x7fc2901093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.549+0000 7fc296600640 1 -- 172.21.15.84:0/900311955 >> 172.21.15.84:0/900311955 conn(0x7fc290077f30 msgr2=0x7fc29007a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:25.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.550+0000 7fc296600640 1 -- 172.21.15.84:0/900311955 shutdown_connections 2024-04-28T12:51:25.547 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.550+0000 7fc296600640 1 -- 172.21.15.84:0/900311955 wait complete. 2024-04-28T12:51:25.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.550+0000 7fc296600640 1 Processor -- start 2024-04-28T12:51:25.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.551+0000 7fc296600640 1 -- start start 2024-04-28T12:51:25.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.551+0000 7fc296600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc290108f90 0x7fc29019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:25.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.551+0000 7fc296600640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc29010a180 0x7fc29019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:25.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.551+0000 7fc296600640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc29010ab40 0x7fc2901a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:25.549 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.551+0000 7fc296600640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc290114b20 con 0x7fc29010ab40 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.551+0000 7fc296600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc2901149a0 con 0x7fc290108f90 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.551+0000 7fc296600640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc290114ca0 con 0x7fc29010a180 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.551+0000 7fc28fe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc29010ab40 0x7fc2901a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.551+0000 7fc28fe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc29010ab40 0x7fc2901a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43088/0 (socket says 172.21.15.84:43088) 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.551+0000 7fc28fe00640 1 -- 172.21.15.84:0/548340084 learned_addr learned my addr 172.21.15.84:0/548340084 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.551+0000 7fc287e00640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc29010a180 0x7fc29019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.551+0000 7fc28f400640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc290108f90 0x7fc29019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.552+0000 7fc28fe00640 1 -- 172.21.15.84:0/548340084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc29010a180 msgr2=0x7fc29019f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.552+0000 7fc28fe00640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc29010a180 0x7fc29019f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.552+0000 7fc28fe00640 1 -- 172.21.15.84:0/548340084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc290108f90 msgr2=0x7fc29019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.552+0000 7fc28fe00640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc290108f90 0x7fc29019f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.552+0000 7fc28fe00640 1 -- 172.21.15.84:0/548340084 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2901a6dc0 con 0x7fc29010ab40 2024-04-28T12:51:25.550 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.552+0000 7fc28f400640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc290108f90 0x7fc29019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:51:25.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.552+0000 7fc28fe00640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc29010ab40 0x7fc2901a6750 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc28800eb10 tx=0x7fc28800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:25.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.552+0000 7fc28cc00640 1 -- 172.21.15.84:0/548340084 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc288004b20 con 0x7fc29010ab40 2024-04-28T12:51:25.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.552+0000 7fc28cc00640 1 -- 172.21.15.84:0/548340084 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc288004cc0 con 0x7fc29010ab40 2024-04-28T12:51:25.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.552+0000 7fc28cc00640 1 -- 172.21.15.84:0/548340084 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc288010670 con 0x7fc29010ab40 2024-04-28T12:51:25.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.552+0000 7fc296600640 1 -- 172.21.15.84:0/548340084 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2901a70b0 con 0x7fc29010ab40 2024-04-28T12:51:25.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.552+0000 7fc296600640 1 -- 172.21.15.84:0/548340084 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2901a7580 con 0x7fc29010ab40 2024-04-28T12:51:25.551 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.553+0000 7fc296600640 1 -- 172.21.15.84:0/548340084 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc290110330 con 0x7fc29010ab40 2024-04-28T12:51:25.552 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.556+0000 7fc28cc00640 1 -- 172.21.15.84:0/548340084 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc288005020 con 0x7fc29010ab40 2024-04-28T12:51:25.553 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.557+0000 7fc28cc00640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc2740a3c40 0x7fc2740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:25.553 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.557+0000 7fc28cc00640 1 -- 172.21.15.84:0/548340084 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fc2880d0ee0 con 0x7fc29010ab40 2024-04-28T12:51:25.553 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.557+0000 7fc28f400640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc2740a3c40 0x7fc2740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:25.554 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.558+0000 7fc28f400640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc2740a3c40 0x7fc2740a6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc290109fe0 tx=0x7fc280005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:25.557 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.561+0000 7fc28cc00640 1 -- 172.21.15.84:0/548340084 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc28809d3e0 con 0x7fc29010ab40 2024-04-28T12:51:25.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.762+0000 7fc296600640 1 -- 172.21.15.84:0/548340084 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc290063560 con 0x7fc29010ab40 2024-04-28T12:51:25.759 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.763+0000 7fc28cc00640 1 -- 172.21.15.84:0/548340084 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fc28809cce0 con 0x7fc29010ab40 2024-04-28T12:51:25.760 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T12:51:25.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.765+0000 7fc296600640 1 -- 172.21.15.84:0/548340084 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc2740a3c40 msgr2=0x7fc2740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.765+0000 7fc296600640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc2740a3c40 0x7fc2740a6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc290109fe0 tx=0x7fc280005f90 comp rx=0 tx=0).stop 2024-04-28T12:51:25.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.765+0000 7fc296600640 1 -- 172.21.15.84:0/548340084 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc29010ab40 msgr2=0x7fc2901a6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.765+0000 7fc296600640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc29010ab40 0x7fc2901a6750 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc28800eb10 tx=0x7fc28800c2d0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.765+0000 7fc296600640 1 -- 172.21.15.84:0/548340084 shutdown_connections 2024-04-28T12:51:25.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.765+0000 7fc296600640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc2740a3c40 0x7fc2740a6100 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.765+0000 7fc296600640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc29010ab40 0x7fc2901a6750 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.765+0000 7fc296600640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc29010a180 0x7fc29019f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.765+0000 7fc296600640 1 --2- 172.21.15.84:0/548340084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc290108f90 0x7fc29019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.765+0000 7fc296600640 1 -- 172.21.15.84:0/548340084 >> 172.21.15.84:0/548340084 conn(0x7fc290077f30 msgr2=0x7fc29010d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:25.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.766+0000 7fc296600640 1 -- 172.21.15.84:0/548340084 shutdown_connections 2024-04-28T12:51:25.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.766+0000 7fc296600640 1 -- 172.21.15.84:0/548340084 wait complete. 2024-04-28T12:51:25.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T12:51:25.981 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.985+0000 7f70cd000640 1 -- 172.21.15.84:0/2327705957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70c8109f80 msgr2=0x7f70c810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.985+0000 7f70cd000640 1 --2- 172.21.15.84:0/2327705957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70c8109f80 0x7f70c810a400 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009900 tx=0x7f70b402a3f0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.985+0000 7f70cd000640 1 -- 172.21.15.84:0/2327705957 shutdown_connections 2024-04-28T12:51:25.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.985+0000 7f70cd000640 1 --2- 172.21.15.84:0/2327705957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70c810a940 0x7f70c81111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.985+0000 7f70cd000640 1 --2- 172.21.15.84:0/2327705957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70c8109f80 0x7f70c810a400 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.985+0000 7f70cd000640 1 --2- 172.21.15.84:0/2327705957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c8108d90 0x7f70c81091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.985+0000 7f70cd000640 1 -- 172.21.15.84:0/2327705957 >> 172.21.15.84:0/2327705957 conn(0x7f70c8077d30 msgr2=0x7f70c807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:25.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.985+0000 7f70cd000640 1 -- 172.21.15.84:0/2327705957 shutdown_connections 2024-04-28T12:51:25.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.985+0000 7f70cd000640 1 -- 172.21.15.84:0/2327705957 wait complete. 2024-04-28T12:51:25.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.986+0000 7f70cd000640 1 Processor -- start 2024-04-28T12:51:25.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.986+0000 7f70cd000640 1 -- start start 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.986+0000 7f70cd000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70c8108d90 0x7f70c819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.986+0000 7f70cd000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c8109f80 0x7f70c819f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.986+0000 7f70cd000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70c810a940 0x7f70c81a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.986+0000 7f70cd000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81148a0 con 0x7f70c810a940 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.986+0000 7f70cd000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f70c8114720 con 0x7f70c8109f80 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.986+0000 7f70cd000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f70c8114a20 con 0x7f70c8108d90 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.987+0000 7f70c7e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70c810a940 0x7f70c81a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.987+0000 7f70c6a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c8109f80 0x7f70c819f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.987+0000 7f70c7e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70c810a940 0x7f70c81a6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43116/0 (socket says 172.21.15.84:43116) 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.987+0000 7f70c7e00640 1 -- 172.21.15.84:0/2195020644 learned_addr learned my addr 172.21.15.84:0/2195020644 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.987+0000 7f70c7400640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70c8108d90 0x7f70c819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.987+0000 7f70c7e00640 1 -- 172.21.15.84:0/2195020644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70c8108d90 msgr2=0x7f70c819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.987+0000 7f70c7e00640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70c8108d90 0x7f70c819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.987+0000 7f70c7e00640 1 -- 172.21.15.84:0/2195020644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c8109f80 msgr2=0x7f70c819f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:25.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.987+0000 7f70c7e00640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c8109f80 0x7f70c819f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:25.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.987+0000 7f70c7e00640 1 -- 172.21.15.84:0/2195020644 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c81a6b90 con 0x7f70c810a940 2024-04-28T12:51:25.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.987+0000 7f70c7e00640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70c810a940 0x7f70c81a6520 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f70c000eb10 tx=0x7f70c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:25.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.988+0000 7f70bfe00640 1 -- 172.21.15.84:0/2195020644 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0004b20 con 0x7f70c810a940 2024-04-28T12:51:25.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.988+0000 7f70bfe00640 1 -- 172.21.15.84:0/2195020644 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70c0004cc0 con 0x7f70c810a940 2024-04-28T12:51:25.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.988+0000 7f70cd000640 1 -- 172.21.15.84:0/2195020644 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c81a6e80 con 0x7f70c810a940 2024-04-28T12:51:25.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.988+0000 7f70bfe00640 1 -- 172.21.15.84:0/2195020644 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0010670 con 0x7f70c810a940 2024-04-28T12:51:25.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.988+0000 7f70cd000640 1 -- 172.21.15.84:0/2195020644 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c81a7340 con 0x7f70c810a940 2024-04-28T12:51:25.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.989+0000 7f70cd000640 1 -- 172.21.15.84:0/2195020644 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70c8110180 con 0x7f70c810a940 2024-04-28T12:51:25.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.991+0000 7f70bfe00640 1 -- 172.21.15.84:0/2195020644 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f70c0005020 con 0x7f70c810a940 2024-04-28T12:51:25.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.992+0000 7f70bfe00640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f70ac0a3cc0 0x7f70ac0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:25.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.992+0000 7f70bfe00640 1 -- 172.21.15.84:0/2195020644 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f70c00d0e50 con 0x7f70c810a940 2024-04-28T12:51:25.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.993+0000 7f70c7400640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f70ac0a3cc0 0x7f70ac0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:25.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.993+0000 7f70c7400640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f70ac0a3cc0 0x7f70ac0a6180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f70c8109de0 tx=0x7f70b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:25.992 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:25.996+0000 7f70bfe00640 1 -- 172.21.15.84:0/2195020644 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f70c009d2f0 con 0x7f70c810a940 2024-04-28T12:51:26.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:25 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2695564487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:51:26.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:25 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/609327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:51:26.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:25 smithi179 ceph-mon[63029]: pgmap v412: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:26.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:25 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/548340084' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:51:26.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:25 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2695564487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:51:26.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:25 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/609327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:51:26.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:25 smithi084 ceph-mon[28081]: pgmap v412: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:26.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:25 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/548340084' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:51:26.235 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.239+0000 7f70cd000640 1 -- 172.21.15.84:0/2195020644 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70c802dad0 con 0x7f70c810a940 2024-04-28T12:51:26.236 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.240+0000 7f70bfe00640 1 -- 172.21.15.84:0/2195020644 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11884 (secure 0 0 0) 0x7f70c009cbf0 con 0x7f70c810a940 2024-04-28T12:51:26.237 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:51:26.237 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6841/1196924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":1196924806},{"type":"v1","addr":"172.21.15.84:6841","nonce":1196924806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:34.334879+0000","modified":"2024-04-28T12:49:08.738419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24562,"mds_2":24545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6839/3812720642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":3812720642},{"type":"v1","addr":"172.21.15.161:6839","nonce":3812720642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24545,24562,24602]},"id":1}]} 2024-04-28T12:51:26.237 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 24 2024-04-28T12:51:26.239 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.242+0000 7f70cd000640 1 -- 172.21.15.84:0/2195020644 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f70ac0a3cc0 msgr2=0x7f70ac0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.242+0000 7f70cd000640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f70ac0a3cc0 0x7f70ac0a6180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f70c8109de0 tx=0x7f70b8005f90 comp rx=0 tx=0).stop 2024-04-28T12:51:26.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.242+0000 7f70cd000640 1 -- 172.21.15.84:0/2195020644 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70c810a940 msgr2=0x7f70c81a6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.242+0000 7f70cd000640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70c810a940 0x7f70c81a6520 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f70c000eb10 tx=0x7f70c000c2d0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.243+0000 7f70cd000640 1 -- 172.21.15.84:0/2195020644 shutdown_connections 2024-04-28T12:51:26.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.243+0000 7f70cd000640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f70ac0a3cc0 0x7f70ac0a6180 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.243+0000 7f70cd000640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70c810a940 0x7f70c81a6520 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.243+0000 7f70cd000640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c8109f80 0x7f70c819f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.243+0000 7f70cd000640 1 --2- 172.21.15.84:0/2195020644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70c8108d90 0x7f70c819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.243+0000 7f70cd000640 1 -- 172.21.15.84:0/2195020644 >> 172.21.15.84:0/2195020644 conn(0x7f70c8077d30 msgr2=0x7f70c810d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:26.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.243+0000 7f70cd000640 1 -- 172.21.15.84:0/2195020644 shutdown_connections 2024-04-28T12:51:26.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.243+0000 7f70cd000640 1 -- 172.21.15.84:0/2195020644 wait complete. 2024-04-28T12:51:26.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:25 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2695564487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T12:51:26.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:25 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/609327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T12:51:26.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:25 smithi161 ceph-mon[62387]: pgmap v412: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:26.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:25 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/548340084' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T12:51:26.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T12:51:26.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.443+0000 7fc304800640 1 -- 172.21.15.84:0/1000763178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2fc104b90 msgr2=0x7fc2fc104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.443+0000 7fc304800640 1 --2- 172.21.15.84:0/1000763178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2fc104b90 0x7fc2fc104fb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009900 tx=0x7fc2f002a3f0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.444+0000 7fc304800640 1 -- 172.21.15.84:0/1000763178 shutdown_connections 2024-04-28T12:51:26.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.444+0000 7fc304800640 1 --2- 172.21.15.84:0/1000763178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2fc106740 0x7fc2fc10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.444+0000 7fc304800640 1 --2- 172.21.15.84:0/1000763178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc105d80 0x7fc2fc106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.444+0000 7fc304800640 1 --2- 172.21.15.84:0/1000763178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2fc104b90 0x7fc2fc104fb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.444+0000 7fc304800640 1 -- 172.21.15.84:0/1000763178 >> 172.21.15.84:0/1000763178 conn(0x7fc2fc100120 msgr2=0x7fc2fc102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:26.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.444+0000 7fc304800640 1 -- 172.21.15.84:0/1000763178 shutdown_connections 2024-04-28T12:51:26.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.444+0000 7fc304800640 1 -- 172.21.15.84:0/1000763178 wait complete. 2024-04-28T12:51:26.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.445+0000 7fc304800640 1 Processor -- start 2024-04-28T12:51:26.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.445+0000 7fc304800640 1 -- start start 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc304800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2fc104b90 0x7fc2fc19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc304800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2fc105d80 0x7fc2fc19b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc304800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc106740 0x7fc2fc1a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc304800640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc1106e0 con 0x7fc2fc104b90 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc304800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc110560 con 0x7fc2fc105d80 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc304800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc110860 con 0x7fc2fc106740 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc301e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2fc104b90 0x7fc2fc19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc302800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc106740 0x7fc2fc1a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc301400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2fc105d80 0x7fc2fc19b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc302800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc106740 0x7fc2fc1a2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:44218/0 (socket says 172.21.15.84:44218) 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc302800640 1 -- 172.21.15.84:0/3977092113 learned_addr learned my addr 172.21.15.84:0/3977092113 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc301400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2fc105d80 0x7fc2fc19b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58886/0 (socket says 172.21.15.84:58886) 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc302800640 1 -- 172.21.15.84:0/3977092113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2fc105d80 msgr2=0x7fc2fc19b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc302800640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2fc105d80 0x7fc2fc19b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc302800640 1 -- 172.21.15.84:0/3977092113 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2fc104b90 msgr2=0x7fc2fc19ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc302800640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2fc104b90 0x7fc2fc19ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.446+0000 7fc302800640 1 -- 172.21.15.84:0/3977092113 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2fc1a29b0 con 0x7fc2fc106740 2024-04-28T12:51:26.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.447+0000 7fc301400640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2fc105d80 0x7fc2fc19b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:26.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.447+0000 7fc302800640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc106740 0x7fc2fc1a2310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800eb10 tx=0x7fc2f800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:26.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.447+0000 7fc2f6a00640 1 -- 172.21.15.84:0/3977092113 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8004b20 con 0x7fc2fc106740 2024-04-28T12:51:26.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.447+0000 7fc2f6a00640 1 -- 172.21.15.84:0/3977092113 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2f8004cc0 con 0x7fc2fc106740 2024-04-28T12:51:26.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.447+0000 7fc2f6a00640 1 -- 172.21.15.84:0/3977092113 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8010670 con 0x7fc2fc106740 2024-04-28T12:51:26.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.447+0000 7fc304800640 1 -- 172.21.15.84:0/3977092113 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2fc1a2ca0 con 0x7fc2fc106740 2024-04-28T12:51:26.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.447+0000 7fc304800640 1 -- 172.21.15.84:0/3977092113 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2fc1a31c0 con 0x7fc2fc106740 2024-04-28T12:51:26.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.448+0000 7fc304800640 1 -- 172.21.15.84:0/3977092113 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2fc10bf30 con 0x7fc2fc106740 2024-04-28T12:51:26.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.450+0000 7fc2f6a00640 1 -- 172.21.15.84:0/3977092113 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc2f8005020 con 0x7fc2fc106740 2024-04-28T12:51:26.449 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.452+0000 7fc2f6a00640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc2e40a3cc0 0x7fc2e40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.449 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.453+0000 7fc2f6a00640 1 -- 172.21.15.84:0/3977092113 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fc2f80d0f30 con 0x7fc2fc106740 2024-04-28T12:51:26.449 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.453+0000 7fc301e00640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc2e40a3cc0 0x7fc2e40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.449 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.453+0000 7fc301e00640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc2e40a3cc0 0x7fc2e40a6180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0005ab0 tx=0x7fc2f0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:26.452 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.456+0000 7fc2f6a00640 1 -- 172.21.15.84:0/3977092113 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc2f809d3d0 con 0x7fc2fc106740 2024-04-28T12:51:26.656 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.659+0000 7fc304800640 1 -- 172.21.15.84:0/3977092113 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2fc02dd20 con 0x7fc2fc106740 2024-04-28T12:51:26.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.662+0000 7fc2f6a00640 1 -- 172.21.15.84:0/3977092113 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24634 (secure 0 0 0) 0x7fc2f809ccd0 con 0x7fc2fc106740 2024-04-28T12:51:26.659 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T12:51:26.659 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":94,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:50:30.848698+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.8","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T12:51:26.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.664+0000 7fc304800640 1 -- 172.21.15.84:0/3977092113 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc2e40a3cc0 msgr2=0x7fc2e40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.664+0000 7fc304800640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc2e40a3cc0 0x7fc2e40a6180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0005ab0 tx=0x7fc2f0005a20 comp rx=0 tx=0).stop 2024-04-28T12:51:26.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.664+0000 7fc304800640 1 -- 172.21.15.84:0/3977092113 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc106740 msgr2=0x7fc2fc1a2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.664+0000 7fc304800640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc106740 0x7fc2fc1a2310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800eb10 tx=0x7fc2f800c2d0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.664+0000 7fc304800640 1 -- 172.21.15.84:0/3977092113 shutdown_connections 2024-04-28T12:51:26.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.664+0000 7fc304800640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc2e40a3cc0 0x7fc2e40a6180 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.664+0000 7fc304800640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc2fc106740 0x7fc2fc1a2310 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.664+0000 7fc304800640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2fc105d80 0x7fc2fc19b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.664+0000 7fc304800640 1 --2- 172.21.15.84:0/3977092113 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2fc104b90 0x7fc2fc19ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.662 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.664+0000 7fc304800640 1 -- 172.21.15.84:0/3977092113 >> 172.21.15.84:0/3977092113 conn(0x7fc2fc100120 msgr2=0x7fc2fc109100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:26.662 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.664+0000 7fc304800640 1 -- 172.21.15.84:0/3977092113 shutdown_connections 2024-04-28T12:51:26.662 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.665+0000 7fc304800640 1 -- 172.21.15.84:0/3977092113 wait complete. 2024-04-28T12:51:26.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-28T12:48:34.334879+0000', 'modified': '2024-04-28T12:49:08.738419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24602, 'mds_1': 24562, 'mds_2': 24545}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24509': {'gid': 24509, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/3348651842', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 3348651842}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 3348651842}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.179:6837/2376856149', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6836', 'nonce': 2376856149}, {'type': 'v1', 'addr': '172.21.15.179:6837', 'nonce': 2376856149}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24544': {'gid': 24544, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.179:6839/3879704230', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6838', 'nonce': 3879704230}, {'type': 'v1', 'addr': '172.21.15.179:6839', 'nonce': 3879704230}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24545': {'gid': 24545, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.161:6837/1346455857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 1346455857}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 1346455857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24562': {'gid': 24562, 'name': 'g', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.84:6839/3772865132', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6838', 'nonce': 3772865132}, {'type': 'v1', 'addr': '172.21.15.84:6839', 'nonce': 3772865132}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24602': {'gid': 24602, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6839/3812720642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 3812720642}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 3812720642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24602, 'qdb_cluster': [24545, 24562, 24602]} 2024-04-28T12:51:26.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-28T12:51:26.680 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-28T12:51:26.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.875+0000 7ff363e00640 1 -- 172.21.15.84:0/2387798016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff364108440 msgr2=0x7ff36410a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.875+0000 7ff363e00640 1 --2- 172.21.15.84:0/2387798016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff364108440 0x7ff36410a890 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009900 tx=0x7ff35c02a410 comp rx=0 tx=0).stop 2024-04-28T12:51:26.872 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.875+0000 7ff363e00640 1 -- 172.21.15.84:0/2387798016 shutdown_connections 2024-04-28T12:51:26.872 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.875+0000 7ff363e00640 1 --2- 172.21.15.84:0/2387798016 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff364108440 0x7ff36410a890 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.872 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.875+0000 7ff363e00640 1 --2- 172.21.15.84:0/2387798016 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364105b10 0x7ff364107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.872 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.875+0000 7ff363e00640 1 --2- 172.21.15.84:0/2387798016 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3641031e0 0x7ff3641055d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.872 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.875+0000 7ff363e00640 1 -- 172.21.15.84:0/2387798016 >> 172.21.15.84:0/2387798016 conn(0x7ff3640fcbb0 msgr2=0x7ff3640ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:26.872 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.875+0000 7ff363e00640 1 -- 172.21.15.84:0/2387798016 shutdown_connections 2024-04-28T12:51:26.872 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.875+0000 7ff363e00640 1 -- 172.21.15.84:0/2387798016 wait complete. 2024-04-28T12:51:26.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.876+0000 7ff363e00640 1 Processor -- start 2024-04-28T12:51:26.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.876+0000 7ff363e00640 1 -- start start 2024-04-28T12:51:26.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.876+0000 7ff363e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3641031e0 0x7ff364198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.876+0000 7ff363e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364105b10 0x7ff364198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.876+0000 7ff363e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff364108440 0x7ff36419ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.876+0000 7ff363e00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff36410e290 con 0x7ff3641031e0 2024-04-28T12:51:26.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.876+0000 7ff363e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff36410e110 con 0x7ff364105b10 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.876+0000 7ff363e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff36410e410 con 0x7ff364108440 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.876+0000 7ff362a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3641031e0 0x7ff364198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.876+0000 7ff362000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364105b10 0x7ff364198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.876+0000 7ff362a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3641031e0 0x7ff364198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43130/0 (socket says 172.21.15.84:43130) 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.877+0000 7ff362a00640 1 -- 172.21.15.84:0/2403673207 learned_addr learned my addr 172.21.15.84:0/2403673207 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.876+0000 7ff362000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364105b10 0x7ff364198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58892/0 (socket says 172.21.15.84:58892) 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.877+0000 7ff363400640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff364108440 0x7ff36419ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.877+0000 7ff362a00640 1 -- 172.21.15.84:0/2403673207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff364108440 msgr2=0x7ff36419ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.877+0000 7ff362a00640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff364108440 0x7ff36419ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.877+0000 7ff362a00640 1 -- 172.21.15.84:0/2403673207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364105b10 msgr2=0x7ff364198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.877+0000 7ff362a00640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364105b10 0x7ff364198ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.877+0000 7ff362a00640 1 -- 172.21.15.84:0/2403673207 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3641a0580 con 0x7ff3641031e0 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.877+0000 7ff362000640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364105b10 0x7ff364198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:26.875 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.877+0000 7ff362a00640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3641031e0 0x7ff364198960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff35400ebc0 tx=0x7ff35400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:26.876 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.877+0000 7ff35b400640 1 -- 172.21.15.84:0/2403673207 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff354010630 con 0x7ff3641031e0 2024-04-28T12:51:26.876 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.878+0000 7ff35b400640 1 -- 172.21.15.84:0/2403673207 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff354010c80 con 0x7ff3641031e0 2024-04-28T12:51:26.876 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.878+0000 7ff363e00640 1 -- 172.21.15.84:0/2403673207 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3641a0870 con 0x7ff3641031e0 2024-04-28T12:51:26.876 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.878+0000 7ff35b400640 1 -- 172.21.15.84:0/2403673207 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff354018b60 con 0x7ff3641031e0 2024-04-28T12:51:26.876 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.878+0000 7ff363e00640 1 -- 172.21.15.84:0/2403673207 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3641a0e10 con 0x7ff3641031e0 2024-04-28T12:51:26.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.882+0000 7ff36ca67740 1 Processor -- start 2024-04-28T12:51:26.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.882+0000 7ff36ca67740 1 -- start start 2024-04-28T12:51:26.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.882+0000 7ff36ca67740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.882+0000 7ff36ca67740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015cb0c0 0x5615015cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.882+0000 7ff36ca67740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015d0230 0x5615015d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.882+0000 7ff36ca67740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x56150152a600 con 0x5615015cb0c0 2024-04-28T12:51:26.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.882+0000 7ff35b400640 1 -- 172.21.15.84:0/2403673207 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff354018d00 con 0x7ff3641031e0 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.882+0000 7ff36ca67740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5615014c7100 con 0x5615015d0230 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.882+0000 7ff36ca67740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5615015265c0 con 0x5615015ca6d0 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.882+0000 7ff363400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.883+0000 7ff363400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:44244/0 (socket says 172.21.15.84:44244) 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.883+0000 7ff363400640 1 -- 172.21.15.84:0/337699931 learned_addr learned my addr 172.21.15.84:0/337699931 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.883+0000 7ff362a00640 1 --2- 172.21.15.84:0/337699931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015cb0c0 0x5615015cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.883+0000 7ff362000640 1 --2- 172.21.15.84:0/337699931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015d0230 0x5615015d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.883+0000 7ff363400640 1 -- 172.21.15.84:0/337699931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015d0230 msgr2=0x5615015d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.883+0000 7ff363400640 1 --2- 172.21.15.84:0/337699931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015d0230 0x5615015d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.883+0000 7ff363400640 1 -- 172.21.15.84:0/337699931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015cb0c0 msgr2=0x5615015cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.883+0000 7ff363400640 1 --2- 172.21.15.84:0/337699931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015cb0c0 0x5615015cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.883+0000 7ff363400640 1 -- 172.21.15.84:0/337699931 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56150123c2d0 con 0x5615015ca6d0 2024-04-28T12:51:26.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.883+0000 7ff362a00640 1 --2- 172.21.15.84:0/337699931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015cb0c0 0x5615015cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:51:26.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.883+0000 7ff363400640 1 --2- 172.21.15.84:0/337699931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015caaf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff35c033000 tx=0x7ff35c005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5856fa960b91cf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:26.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.884+0000 7ff34fe00640 1 -- 172.21.15.84:0/337699931 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c035030 con 0x5615015ca6d0 2024-04-28T12:51:26.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.884+0000 7ff34fe00640 1 -- 172.21.15.84:0/337699931 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff35c040070 con 0x5615015ca6d0 2024-04-28T12:51:26.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.884+0000 7ff34fe00640 1 -- 172.21.15.84:0/337699931 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c003930 con 0x5615015ca6d0 2024-04-28T12:51:26.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.884+0000 7ff35b400640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff3480a3cc0 0x7ff3480a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.884+0000 7ff35b400640 1 -- 172.21.15.84:0/2403673207 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff3540d0eb0 con 0x7ff3641031e0 2024-04-28T12:51:26.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.884+0000 7ff362a00640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff3480a3cc0 0x7ff3480a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.884+0000 7ff36ca67740 1 -- 172.21.15.84:0/337699931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 msgr2=0x5615015caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.884+0000 7ff36ca67740 1 --2- 172.21.15.84:0/337699931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015caaf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff35c033000 tx=0x7ff35c005dd0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.885+0000 7ff36ca67740 1 -- 172.21.15.84:0/337699931 shutdown_connections 2024-04-28T12:51:26.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.885+0000 7ff36ca67740 1 --2- 172.21.15.84:0/337699931 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015d0230 0x5615015d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.885+0000 7ff36ca67740 1 --2- 172.21.15.84:0/337699931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015cb0c0 0x5615015cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.885+0000 7ff36ca67740 1 --2- 172.21.15.84:0/337699931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015caaf0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.885+0000 7ff36ca67740 1 -- 172.21.15.84:0/337699931 >> 172.21.15.84:0/337699931 conn(0x56150155ad10 msgr2=0x5615015c1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:26.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.885+0000 7ff36ca67740 1 -- 172.21.15.84:0/337699931 shutdown_connections 2024-04-28T12:51:26.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.885+0000 7ff362a00640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff3480a3cc0 0x7ff3480a6180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x5615015cb4c0 tx=0x7ff354000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:26.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.885+0000 7ff36ca67740 1 -- 172.21.15.84:0/337699931 wait complete. 2024-04-28T12:51:26.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.886+0000 7ff36ca67740 1 -- start start 2024-04-28T12:51:26.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.886+0000 7ff36ca67740 1 Processor -- start 2024-04-28T12:51:26.886 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.886+0000 7ff36ca67740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015ca6d0 0x5615015d4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.886 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.886+0000 7ff36ca67740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015cb0c0 0x5615015d5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.886 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.886+0000 7ff36ca67740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015d0230 0x5615015dc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.886 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.886+0000 7ff36ca67740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x5615015d3180 con 0x5615015d0230 2024-04-28T12:51:26.886 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.886+0000 7ff36ca67740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5615014b1f70 con 0x5615015ca6d0 2024-04-28T12:51:26.886 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.886+0000 7ff36ca67740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5615015d3300 con 0x5615015cb0c0 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.886+0000 7ff36ca67740 10 client.0 _ll_register_callbacks cb 0x561501520480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff362000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015cb0c0 0x5615015d5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff362000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015cb0c0 0x5615015d5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:44250/0 (socket says 172.21.15.84:44250) 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff362000640 1 -- 172.21.15.84:0/1411508467 learned_addr learned my addr 172.21.15.84:0/1411508467 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff363400640 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015ca6d0 0x5615015d4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff363400640 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015d0230 0x5615015dc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff362000640 1 -- 172.21.15.84:0/1411508467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015ca6d0 msgr2=0x5615015d4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff362000640 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015ca6d0 0x5615015d4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff362000640 1 -- 172.21.15.84:0/1411508467 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015d0230 msgr2=0x5615015dc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff362000640 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015d0230 0x5615015dc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff362000640 1 -- 172.21.15.84:0/1411508467 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615013066b0 con 0x5615015cb0c0 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff363400640 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015ca6d0 0x5615015d4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff362000640 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015cb0c0 0x5615015d5030 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff350012d20 tx=0x7ff35000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81a3f09add84ecae server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.887+0000 7ff34cc00640 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-04-28T12:51:26.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.888+0000 7ff34cc00640 1 -- 172.21.15.84:0/1411508467 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff350012110 con 0x5615015cb0c0 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.888+0000 7ff34cc00640 1 -- 172.21.15.84:0/1411508467 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff350004050 con 0x5615015cb0c0 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.888+0000 7ff36ca67740 1 -- 172.21.15.84:0/1411508467 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff35c040070 con 0x5615015cb0c0 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.888+0000 7ff34cc00640 1 -- 172.21.15.84:0/1411508467 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff350025050 con 0x5615015cb0c0 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.888+0000 7ff36ca67740 10 client.25330 fetch_fsmap learned FSMap version 24 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.888+0000 7ff36ca67740 1 -- 172.21.15.84:0/1411508467 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5615015df120 con 0x5615015cb0c0 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.888+0000 7ff34cc00640 1 -- 172.21.15.84:0/1411508467 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff35c040070 con 0x5615015cb0c0 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.889+0000 7ff34cc00640 1 -- 172.21.15.84:0/1411508467 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff350003cd0 con 0x5615015cb0c0 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.889+0000 7ff36ca67740 10 client.25330 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.889+0000 7ff36ca67740 10 client.25330 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.889+0000 7ff36ca67740 20 client.25330 populate_metadata read hostname 'smithi084' 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.889+0000 7ff36ca67740 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x5615015e1800 0x5615015e2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.889+0000 7ff36ca67740 4 client.25330 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.889+0000 7ff36ca67740 1 -- 172.21.15.84:0/1411508467 --> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561501538210 con 0x5615015e1800 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.889+0000 7ff363400640 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x5615015e1800 0x5615015e2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.890+0000 7ff34cc00640 10 client.25330 ms_handle_connect on v2:172.21.15.161:6836/1346455857 2024-04-28T12:51:26.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.890+0000 7ff363400640 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x5615015e1800 0x5615015e2c70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3fc615bfcc644e37 server_cookie=76c548694920f0 in_seq=0 out_seq=0 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.890+0000 7ff34cc00640 1 -- 172.21.15.84:0/1411508467 <== mds.2 v2:172.21.15.161:6836/1346455857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561501538210 con 0x5615015e1800 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.890+0000 7ff34cc00640 10 client.25330 handle_command_reply: tid=0 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff36ca67740 1 client.25330 shutdown 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff36ca67740 1 -- 172.21.15.84:0/1411508467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015cb0c0 msgr2=0x5615015d5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff36ca67740 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015cb0c0 0x5615015d5030 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff350012d20 tx=0x7ff35000f790 comp rx=0 tx=0).stop 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff36ca67740 1 -- 172.21.15.84:0/1411508467 shutdown_connections 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff36ca67740 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x5615015e1800 0x5615015e2c70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff36ca67740 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015d0230 0x5615015dc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff36ca67740 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015cb0c0 0x5615015d5030 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff36ca67740 1 --2- 172.21.15.84:0/1411508467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015ca6d0 0x5615015d4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff34cc00640 0 client.25330 ms_handle_reset on v2:172.21.15.161:6836/1346455857 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff36ca67740 1 -- 172.21.15.84:0/1411508467 >> 172.21.15.84:0/1411508467 conn(0x56150155ad10 msgr2=0x5615015c0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:26.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff36ca67740 1 -- 172.21.15.84:0/1411508467 shutdown_connections 2024-04-28T12:51:26.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff36ca67740 1 -- 172.21.15.84:0/1411508467 wait complete. 2024-04-28T12:51:26.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.891+0000 7ff36ca67740 20 client.25330 trim_cache size 0 max 16384 2024-04-28T12:51:26.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.912+0000 7ff36ca67740 1 Processor -- start 2024-04-28T12:51:26.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.912+0000 7ff36ca67740 1 -- start start 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.912+0000 7ff36ca67740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.912+0000 7ff36ca67740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015cb0c0 0x5615015dc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.912+0000 7ff36ca67740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015d0230 0x5615015d4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.912+0000 7ff36ca67740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x5615015e0fb0 con 0x5615015d0230 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.912+0000 7ff36ca67740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5615015e1680 con 0x5615015cb0c0 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.912+0000 7ff36ca67740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5615015e1d50 con 0x5615015ca6d0 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.913+0000 7ff362000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015cb0c0 0x5615015dc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.913+0000 7ff363400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.913+0000 7ff362000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015cb0c0 0x5615015dc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58924/0 (socket says 172.21.15.84:58924) 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.913+0000 7ff362000640 1 -- 172.21.15.84:0/3601102203 learned_addr learned my addr 172.21.15.84:0/3601102203 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.913+0000 7ff363400640 1 --2- 172.21.15.84:0/3601102203 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015d0230 0x5615015d4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.913+0000 7ff362000640 1 -- 172.21.15.84:0/3601102203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 msgr2=0x5615015cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.913+0000 7ff362000640 1 --2- 172.21.15.84:0/3601102203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.913+0000 7ff362000640 1 -- 172.21.15.84:0/3601102203 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015d0230 msgr2=0x5615015d4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:26.911 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.913+0000 7ff362000640 1 --2- 172.21.15.84:0/3601102203 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015d0230 0x5615015d4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.911 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.913+0000 7ff362000640 1 -- 172.21.15.84:0/3601102203 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615015d5ae0 con 0x5615015cb0c0 2024-04-28T12:51:26.911 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.913+0000 7ff362000640 1 --2- 172.21.15.84:0/3601102203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015cb0c0 0x5615015dc030 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff350000c00 tx=0x7ff3500123d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b115554d1ae9dfe server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:26.913 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.914+0000 7ff34f400640 1 -- 172.21.15.84:0/3601102203 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35000fe80 con 0x5615015cb0c0 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.914+0000 7ff34f400640 1 -- 172.21.15.84:0/3601102203 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff350005560 con 0x5615015cb0c0 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.914+0000 7ff34f400640 1 -- 172.21.15.84:0/3601102203 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3500035f0 con 0x5615015cb0c0 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.914+0000 7ff36ca67740 1 -- 172.21.15.84:0/3601102203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015cb0c0 msgr2=0x5615015dc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.914+0000 7ff36ca67740 1 --2- 172.21.15.84:0/3601102203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015cb0c0 0x5615015dc030 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff350000c00 tx=0x7ff3500123d0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.914+0000 7ff36ca67740 1 -- 172.21.15.84:0/3601102203 shutdown_connections 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.914+0000 7ff36ca67740 1 --2- 172.21.15.84:0/3601102203 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015d0230 0x5615015d4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.914+0000 7ff36ca67740 1 --2- 172.21.15.84:0/3601102203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015cb0c0 0x5615015dc030 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.914+0000 7ff36ca67740 1 --2- 172.21.15.84:0/3601102203 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.914+0000 7ff36ca67740 1 -- 172.21.15.84:0/3601102203 >> 172.21.15.84:0/3601102203 conn(0x56150155ad10 msgr2=0x5615015e05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.915+0000 7ff36ca67740 1 -- 172.21.15.84:0/3601102203 shutdown_connections 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.915+0000 7ff36ca67740 1 -- 172.21.15.84:0/3601102203 wait complete. 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.915+0000 7ff36ca67740 1 -- start start 2024-04-28T12:51:26.914 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.915+0000 7ff36ca67740 1 Processor -- start 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff36ca67740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015d77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff36ca67740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015cb0c0 0x5615015d7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff36ca67740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015d0230 0x5615015dad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff36ca67740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x5615015e2af0 con 0x5615015cb0c0 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff36ca67740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5615015e2420 con 0x5615015d0230 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff36ca67740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x5615015e31c0 con 0x5615015ca6d0 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff36ca67740 10 client.0 _ll_register_callbacks cb 0x7ff350009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff363400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015d77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff363400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015d77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:44276/0 (socket says 172.21.15.84:44276) 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff363400640 1 -- 172.21.15.84:0/2876936682 learned_addr learned my addr 172.21.15.84:0/2876936682 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff362000640 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015cb0c0 0x5615015d7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff363400640 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015d0230 0x5615015dad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff362000640 1 -- 172.21.15.84:0/2876936682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 msgr2=0x5615015d77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff362000640 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015d77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff362000640 1 -- 172.21.15.84:0/2876936682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015d0230 msgr2=0x5615015dad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:26.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff362000640 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015d0230 0x5615015dad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff362000640 1 -- 172.21.15.84:0/2876936682 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615015db340 con 0x5615015cb0c0 2024-04-28T12:51:26.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.916+0000 7ff363400640 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015d77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:26.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.917+0000 7ff362000640 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015cb0c0 0x5615015d7d30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff3500106a0 tx=0x7ff350010950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b2486dcc9c5160 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:26.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.917+0000 7ff34cc00640 10 client.0 ms_handle_connect on v2:172.21.15.84:3300/0 2024-04-28T12:51:26.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.917+0000 7ff34cc00640 1 -- 172.21.15.84:0/2876936682 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35000fe80 con 0x5615015cb0c0 2024-04-28T12:51:26.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.917+0000 7ff34cc00640 1 -- 172.21.15.84:0/2876936682 <== mon.0 v2:172.21.15.84:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff35002f070 con 0x5615015cb0c0 2024-04-28T12:51:26.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.917+0000 7ff36ca67740 1 -- 172.21.15.84:0/2876936682 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff350005560 con 0x5615015cb0c0 2024-04-28T12:51:26.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.917+0000 7ff34cc00640 1 -- 172.21.15.84:0/2876936682 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff350005320 con 0x5615015cb0c0 2024-04-28T12:51:26.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.917+0000 7ff36ca67740 10 client.15228 fetch_fsmap learned FSMap version 24 2024-04-28T12:51:26.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.917+0000 7ff36ca67740 1 -- 172.21.15.84:0/2876936682 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5615016477b0 con 0x5615015cb0c0 2024-04-28T12:51:26.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.917+0000 7ff34cc00640 1 -- 172.21.15.84:0/2876936682 <== mon.0 v2:172.21.15.84:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff350005560 con 0x5615015cb0c0 2024-04-28T12:51:26.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.918+0000 7ff34cc00640 1 -- 172.21.15.84:0/2876936682 <== mon.0 v2:172.21.15.84:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff35002a440 con 0x5615015cb0c0 2024-04-28T12:51:26.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.918+0000 7ff36ca67740 10 client.15228 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T12:51:26.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.918+0000 7ff36ca67740 10 client.15228 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-28T12:51:26.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.918+0000 7ff36ca67740 20 client.15228 populate_metadata read hostname 'smithi084' 2024-04-28T12:51:26.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.918+0000 7ff36ca67740 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x56150164c120 0x561501648c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:26.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.918+0000 7ff36ca67740 4 client.15228 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:51:26.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.918+0000 7ff36ca67740 1 -- 172.21.15.84:0/2876936682 --> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] -- command(tid 0: {"prefix": "status"}) -- 0x561501532080 con 0x56150164c120 2024-04-28T12:51:26.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.918+0000 7ff363400640 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x56150164c120 0x561501648c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff34cc00640 10 client.15228 ms_handle_connect on v2:172.21.15.161:6836/1346455857 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff363400640 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x56150164c120 0x561501648c30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4c592b4032e99569 server_cookie=28672ed7faccbf2e in_seq=0 out_seq=0 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff34cc00640 1 -- 172.21.15.84:0/2876936682 <== mds.2 v2:172.21.15.161:6836/1346455857 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x561501532080 con 0x56150164c120 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff34cc00640 10 client.15228 handle_command_reply: tid=0 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff36ca67740 1 client.15228 shutdown 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff36ca67740 1 -- 172.21.15.84:0/2876936682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015cb0c0 msgr2=0x5615015d7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff36ca67740 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015cb0c0 0x5615015d7d30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff3500106a0 tx=0x7ff350010950 comp rx=0 tx=0).stop 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff36ca67740 1 -- 172.21.15.84:0/2876936682 shutdown_connections 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff36ca67740 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.161:6836/1346455857,v1:172.21.15.161:6837/1346455857] conn(0x56150164c120 0x561501648c30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff36ca67740 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5615015d0230 0x5615015dad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff36ca67740 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x5615015cb0c0 0x5615015d7d30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff34cc00640 0 client.15228 ms_handle_reset on v2:172.21.15.161:6836/1346455857 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff36ca67740 1 --2- 172.21.15.84:0/2876936682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5615015ca6d0 0x5615015d77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff36ca67740 1 -- 172.21.15.84:0/2876936682 >> 172.21.15.84:0/2876936682 conn(0x56150155ad10 msgr2=0x5615015d4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:26.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff36ca67740 1 -- 172.21.15.84:0/2876936682 shutdown_connections 2024-04-28T12:51:26.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.919+0000 7ff36ca67740 1 -- 172.21.15.84:0/2876936682 wait complete. 2024-04-28T12:51:26.920 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff36ca67740 20 client.15228 trim_cache size 0 max 16384 2024-04-28T12:51:26.920 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-28T12:51:26.920 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_fsid": "ddee604a-055b-11ef-bc93-c7b262605968", 2024-04-28T12:51:26.920 INFO:teuthology.orchestra.run.smithi084.stdout: "whoami": 2, 2024-04-28T12:51:26.920 INFO:teuthology.orchestra.run.smithi084.stdout: "id": 24545, 2024-04-28T12:51:26.920 INFO:teuthology.orchestra.run.smithi084.stdout: "want_state": "up:active", 2024-04-28T12:51:26.920 INFO:teuthology.orchestra.run.smithi084.stdout: "state": "up:active", 2024-04-28T12:51:26.920 INFO:teuthology.orchestra.run.smithi084.stdout: "fs_name": "cephfs", 2024-04-28T12:51:26.920 INFO:teuthology.orchestra.run.smithi084.stdout: "rank_uptime": 139.191079293, 2024-04-28T12:51:26.921 INFO:teuthology.orchestra.run.smithi084.stdout: "mdsmap_epoch": 24, 2024-04-28T12:51:26.921 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch": 94, 2024-04-28T12:51:26.921 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch_barrier": 90, 2024-04-28T12:51:26.921 INFO:teuthology.orchestra.run.smithi084.stdout: "uptime": 192.576632136 2024-04-28T12:51:26.921 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-28T12:51:26.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff363e00640 1 -- 172.21.15.84:0/2403673207 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff3480a3cc0 msgr2=0x7ff3480a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff363e00640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff3480a3cc0 0x7ff3480a6180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x5615015cb4c0 tx=0x7ff354000f40 comp rx=0 tx=0).stop 2024-04-28T12:51:26.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff363e00640 1 -- 172.21.15.84:0/2403673207 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3641031e0 msgr2=0x7ff364198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:26.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff363e00640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3641031e0 0x7ff364198960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff35400ebc0 tx=0x7ff35400c6a0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff363e00640 1 -- 172.21.15.84:0/2403673207 shutdown_connections 2024-04-28T12:51:26.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff363e00640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff3480a3cc0 0x7ff3480a6180 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff363e00640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff364108440 0x7ff36419ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff363e00640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff364105b10 0x7ff364198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff363e00640 1 --2- 172.21.15.84:0/2403673207 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3641031e0 0x7ff364198960 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:26.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff363e00640 1 -- 172.21.15.84:0/2403673207 >> 172.21.15.84:0/2403673207 conn(0x7ff3640fcbb0 msgr2=0x7ff364103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:26.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff363e00640 1 -- 172.21.15.84:0/2403673207 shutdown_connections 2024-04-28T12:51:26.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:26.920+0000 7ff363e00640 1 -- 172.21.15.84:0/2403673207 wait complete. 2024-04-28T12:51:26.931 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:26 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2195020644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:51:26.931 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:26 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:51:26.932 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:26 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3977092113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:51:26.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-28T12:51:27.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.118+0000 7ff7b5400640 1 -- 172.21.15.84:0/2606429568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b0076930 msgr2=0x7ff7b0074d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.118+0000 7ff7b5400640 1 --2- 172.21.15.84:0/2606429568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b0076930 0x7ff7b0074d80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009860 tx=0x7ff7a002a340 comp rx=0 tx=0).stop 2024-04-28T12:51:27.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.118+0000 7ff7b5400640 1 -- 172.21.15.84:0/2606429568 shutdown_connections 2024-04-28T12:51:27.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.118+0000 7ff7b5400640 1 --2- 172.21.15.84:0/2606429568 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7b010a610 0x7ff7b010ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.118+0000 7ff7b5400640 1 --2- 172.21.15.84:0/2606429568 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7b0075350 0x7ff7b00757d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.115 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.118+0000 7ff7b5400640 1 --2- 172.21.15.84:0/2606429568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b0076930 0x7ff7b0074d80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.116 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.118+0000 7ff7b5400640 1 -- 172.21.15.84:0/2606429568 >> 172.21.15.84:0/2606429568 conn(0x7ff7b00fdcb0 msgr2=0x7ff7b0100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:27.116 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.118+0000 7ff7b5400640 1 -- 172.21.15.84:0/2606429568 shutdown_connections 2024-04-28T12:51:27.116 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.118+0000 7ff7b5400640 1 -- 172.21.15.84:0/2606429568 wait complete. 2024-04-28T12:51:27.117 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.119+0000 7ff7b5400640 1 Processor -- start 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.119+0000 7ff7b5400640 1 -- start start 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b5400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7b0075350 0x7ff7b019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b5400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7b0076930 0x7ff7b019b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b5400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b010a610 0x7ff7b01a2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b5400640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b0110450 con 0x7ff7b0075350 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b5400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b01102d0 con 0x7ff7b010a610 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b5400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b01105d0 con 0x7ff7b0076930 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7afe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7b0075350 0x7ff7b019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7af400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7b0076930 0x7ff7b019b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b4a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b010a610 0x7ff7b01a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7afe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7b0075350 0x7ff7b019aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43172/0 (socket says 172.21.15.84:43172) 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b4a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b010a610 0x7ff7b01a2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58956/0 (socket says 172.21.15.84:58956) 2024-04-28T12:51:27.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7afe00640 1 -- 172.21.15.84:0/3982405246 learned_addr learned my addr 172.21.15.84:0/3982405246 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:27.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7af400640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7b0076930 0x7ff7b019b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:44300/0 (socket says 172.21.15.84:44300) 2024-04-28T12:51:27.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b4a00640 1 -- 172.21.15.84:0/3982405246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7b0076930 msgr2=0x7ff7b019b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b4a00640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7b0076930 0x7ff7b019b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b4a00640 1 -- 172.21.15.84:0/3982405246 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7b0075350 msgr2=0x7ff7b019aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b4a00640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7b0075350 0x7ff7b019aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.120+0000 7ff7b4a00640 1 -- 172.21.15.84:0/3982405246 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7b01a2770 con 0x7ff7b010a610 2024-04-28T12:51:27.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.121+0000 7ff7b4a00640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b010a610 0x7ff7b01a2100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8009fd0 tx=0x7ff7a800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:27.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.121+0000 7ff7acc00640 1 -- 172.21.15.84:0/3982405246 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a800cd60 con 0x7ff7b010a610 2024-04-28T12:51:27.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.121+0000 7ff7b5400640 1 -- 172.21.15.84:0/3982405246 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7b01a2a60 con 0x7ff7b010a610 2024-04-28T12:51:27.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.121+0000 7ff7acc00640 1 -- 172.21.15.84:0/3982405246 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7a8005020 con 0x7ff7b010a610 2024-04-28T12:51:27.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.121+0000 7ff7acc00640 1 -- 172.21.15.84:0/3982405246 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8010690 con 0x7ff7b010a610 2024-04-28T12:51:27.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.121+0000 7ff7b5400640 1 -- 172.21.15.84:0/3982405246 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7b01a2f60 con 0x7ff7b010a610 2024-04-28T12:51:27.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.124+0000 7ff7b9832740 1 Processor -- start 2024-04-28T12:51:27.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.124+0000 7ff7acc00640 1 -- 172.21.15.84:0/3982405246 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff7a80054d0 con 0x7ff7b010a610 2024-04-28T12:51:27.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.124+0000 7ff7b9832740 1 -- start start 2024-04-28T12:51:27.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7b9832740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317b986e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7b9832740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317b9d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7b9832740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b9de20 0x557317ba02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7b9832740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x557317afc600 con 0x557317b9de20 2024-04-28T12:51:27.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7b9832740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557317a99100 con 0x557317b98cb0 2024-04-28T12:51:27.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7b9832740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x557317af85c0 con 0x557317b982c0 2024-04-28T12:51:27.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7afe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317b9d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7afe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317b9d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58960/0 (socket says 172.21.15.84:58960) 2024-04-28T12:51:27.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7afe00640 1 -- 172.21.15.84:0/1444538947 learned_addr learned my addr 172.21.15.84:0/1444538947 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:27.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7af400640 1 --2- 172.21.15.84:0/1444538947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317b986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7af400640 1 --2- 172.21.15.84:0/1444538947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b9de20 0x557317ba02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7afe00640 1 -- 172.21.15.84:0/1444538947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 msgr2=0x557317b986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7afe00640 1 --2- 172.21.15.84:0/1444538947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317b986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7afe00640 1 -- 172.21.15.84:0/1444538947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b9de20 msgr2=0x557317ba02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7afe00640 1 --2- 172.21.15.84:0/1444538947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b9de20 0x557317ba02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7afe00640 1 -- 172.21.15.84:0/1444538947 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55731780e2d0 con 0x557317b98cb0 2024-04-28T12:51:27.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.125+0000 7ff7af400640 1 --2- 172.21.15.84:0/1444538947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b9de20 0x557317ba02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-28T12:51:27.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.126+0000 7ff7afe00640 1 --2- 172.21.15.84:0/1444538947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317b9d7b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0035040 tx=0x7ff7a0009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ac30de42c7e5f11 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:27.126 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.126+0000 7ff7acc00640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7940a3c40 0x7ff7940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.126 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.126+0000 7ff7a5200640 1 -- 172.21.15.84:0/1444538947 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0037040 con 0x557317b98cb0 2024-04-28T12:51:27.126 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.126+0000 7ff7a5200640 1 -- 172.21.15.84:0/1444538947 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7a0004030 con 0x557317b98cb0 2024-04-28T12:51:27.126 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.126+0000 7ff7a5200640 1 -- 172.21.15.84:0/1444538947 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0003b50 con 0x557317b98cb0 2024-04-28T12:51:27.126 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.126+0000 7ff7acc00640 1 -- 172.21.15.84:0/3982405246 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff7a80d0fe0 con 0x7ff7b010a610 2024-04-28T12:51:27.126 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.126+0000 7ff7af400640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7940a3c40 0x7ff7940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.126 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.127+0000 7ff7b9832740 1 -- 172.21.15.84:0/1444538947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 msgr2=0x557317b9d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.126 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.127+0000 7ff7b9832740 1 --2- 172.21.15.84:0/1444538947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317b9d7b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0035040 tx=0x7ff7a0009510 comp rx=0 tx=0).stop 2024-04-28T12:51:27.126 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.127+0000 7ff7af400640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7940a3c40 0x7ff7940a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x557317b99510 tx=0x7ff79c013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:27.126 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.127+0000 7ff7b9832740 1 -- 172.21.15.84:0/1444538947 shutdown_connections 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.127+0000 7ff7b9832740 1 --2- 172.21.15.84:0/1444538947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b9de20 0x557317ba02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.127+0000 7ff7b9832740 1 --2- 172.21.15.84:0/1444538947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317b9d7b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.127+0000 7ff7b9832740 1 --2- 172.21.15.84:0/1444538947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317b986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.127+0000 7ff7b9832740 1 -- 172.21.15.84:0/1444538947 >> 172.21.15.84:0/1444538947 conn(0x557317b2cd10 msgr2=0x557317b93d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.127+0000 7ff7b9832740 1 -- 172.21.15.84:0/1444538947 shutdown_connections 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.127+0000 7ff7b9832740 1 -- 172.21.15.84:0/1444538947 wait complete. 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.128+0000 7ff7b9832740 1 -- start start 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.128+0000 7ff7b9832740 1 Processor -- start 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.128+0000 7ff7b9832740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317ba2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.128+0000 7ff7b9832740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b98cb0 0x557317ba2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.128+0000 7ff7b9832740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b9de20 0x557317ba9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.128+0000 7ff7b9832740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x557317ba0d70 con 0x557317b98cb0 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.128+0000 7ff7b9832740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557317a83f70 con 0x557317b9de20 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.128+0000 7ff7b9832740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x557317ba0ef0 con 0x557317b982c0 2024-04-28T12:51:27.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff7b9832740 10 client.0 _ll_register_callbacks cb 0x557317af2480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff7b4a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b9de20 0x557317ba9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff7af400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b98cb0 0x557317ba2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff7b4a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b9de20 0x557317ba9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58970/0 (socket says 172.21.15.84:58970) 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff7b4a00640 1 -- 172.21.15.84:0/4064616424 learned_addr learned my addr 172.21.15.84:0/4064616424 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff7afe00640 1 --2- 172.21.15.84:0/4064616424 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317ba2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff7b4a00640 1 -- 172.21.15.84:0/4064616424 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 msgr2=0x557317ba2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff7b4a00640 1 --2- 172.21.15.84:0/4064616424 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317ba2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff7b4a00640 1 -- 172.21.15.84:0/4064616424 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b98cb0 msgr2=0x557317ba2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff7b4a00640 1 --2- 172.21.15.84:0/4064616424 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b98cb0 0x557317ba2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff7b4a00640 1 -- 172.21.15.84:0/4064616424 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573178d86b0 con 0x557317b9de20 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff7b4a00640 1 --2- 172.21.15.84:0/4064616424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b9de20 0x557317ba9c90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff7a801a040 tx=0x7ff7a809cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48472b4acfbaf3c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.129+0000 7ff799600640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.130+0000 7ff799600640 1 -- 172.21.15.84:0/4064616424 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a80a55c0 con 0x557317b9de20 2024-04-28T12:51:27.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.130+0000 7ff799600640 1 -- 172.21.15.84:0/4064616424 <== mon.2 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff7a80a5760 con 0x557317b9de20 2024-04-28T12:51:27.129 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.130+0000 7ff7b9832740 1 -- 172.21.15.84:0/4064616424 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7a0004030 con 0x557317b9de20 2024-04-28T12:51:27.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.130+0000 7ff799600640 1 -- 172.21.15.84:0/4064616424 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff7a80b6200 con 0x557317b9de20 2024-04-28T12:51:27.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.130+0000 7ff7b9832740 10 client.25361 fetch_fsmap learned FSMap version 24 2024-04-28T12:51:27.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.130+0000 7ff7b9832740 1 -- 172.21.15.84:0/4064616424 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557317bacce0 con 0x557317b9de20 2024-04-28T12:51:27.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.130+0000 7ff799600640 1 -- 172.21.15.84:0/4064616424 <== mon.2 v2:172.21.15.161:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7a0004030 con 0x557317b9de20 2024-04-28T12:51:27.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.130+0000 7ff799600640 1 -- 172.21.15.84:0/4064616424 <== mon.2 v2:172.21.15.161:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff7a809bda0 con 0x557317b9de20 2024-04-28T12:51:27.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.130+0000 7ff7b9832740 10 client.25361 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T12:51:27.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.130+0000 7ff7b9832740 10 client.25361 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-28T12:51:27.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.130+0000 7ff7b9832740 20 client.25361 populate_metadata read hostname 'smithi084' 2024-04-28T12:51:27.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.130+0000 7ff7b9832740 1 --2- 172.21.15.84:0/4064616424 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x557317baf330 0x557317bb07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.131+0000 7ff7b9832740 4 client.25361 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:51:27.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.131+0000 7ff7b9832740 1 -- 172.21.15.84:0/4064616424 --> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557317b0a210 con 0x557317baf330 2024-04-28T12:51:27.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.131+0000 7ff7afe00640 1 --2- 172.21.15.84:0/4064616424 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x557317baf330 0x557317bb07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.131+0000 7ff7afe00640 1 --2- 172.21.15.84:0/4064616424 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x557317baf330 0x557317bb07a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c1e567d37074535e server_cookie=a36ff484887f30a3 in_seq=0 out_seq=0 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.131+0000 7ff799600640 10 client.25361 ms_handle_connect on v2:172.21.15.84:6838/3772865132 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.132+0000 7ff799600640 1 -- 172.21.15.84:0/4064616424 <== mds.1 v2:172.21.15.84:6838/3772865132 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557317b0a210 con 0x557317baf330 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.132+0000 7ff799600640 10 client.25361 handle_command_reply: tid=0 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.132+0000 7ff7b9832740 1 client.25361 shutdown 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.132+0000 7ff7b9832740 1 -- 172.21.15.84:0/4064616424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b9de20 msgr2=0x557317ba9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.132+0000 7ff7b9832740 1 --2- 172.21.15.84:0/4064616424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b9de20 0x557317ba9c90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff7a801a040 tx=0x7ff7a809cf20 comp rx=0 tx=0).stop 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.132+0000 7ff7b9832740 1 -- 172.21.15.84:0/4064616424 shutdown_connections 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.132+0000 7ff7b9832740 1 --2- 172.21.15.84:0/4064616424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b9de20 0x557317ba9c90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.132+0000 7ff7b9832740 1 --2- 172.21.15.84:0/4064616424 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b98cb0 0x557317ba2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.132+0000 7ff7b9832740 1 --2- 172.21.15.84:0/4064616424 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x557317baf330 0x557317bb07a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.132+0000 7ff7b9832740 1 --2- 172.21.15.84:0/4064616424 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317ba2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.133+0000 7ff7b9832740 1 -- 172.21.15.84:0/4064616424 >> 172.21.15.84:0/4064616424 conn(0x557317b2cd10 msgr2=0x557317b92f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.133+0000 7ff799600640 0 client.25361 ms_handle_reset on v2:172.21.15.84:6838/3772865132 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.133+0000 7ff7b9832740 1 -- 172.21.15.84:0/4064616424 shutdown_connections 2024-04-28T12:51:27.132 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.133+0000 7ff7b9832740 1 -- 172.21.15.84:0/4064616424 wait complete. 2024-04-28T12:51:27.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.133+0000 7ff7b9832740 20 client.25361 trim_cache size 0 max 16384 2024-04-28T12:51:27.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.156+0000 7ff7b9832740 1 Processor -- start 2024-04-28T12:51:27.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.156+0000 7ff7b9832740 1 -- start start 2024-04-28T12:51:27.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7b9832740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317b8edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7b9832740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317ba9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7b9832740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b9de20 0x557317b9a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7b9832740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x557317baeae0 con 0x557317b9de20 2024-04-28T12:51:27.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7b9832740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557317baf1b0 con 0x557317b98cb0 2024-04-28T12:51:27.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7b9832740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x557317baf880 con 0x557317b982c0 2024-04-28T12:51:27.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7b4a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b9de20 0x557317b9a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7af400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317ba9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7afe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317b8edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7af400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317ba9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58976/0 (socket says 172.21.15.84:58976) 2024-04-28T12:51:27.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7afe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317b8edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:44322/0 (socket says 172.21.15.84:44322) 2024-04-28T12:51:27.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7b4a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b9de20 0x557317b9a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43198/0 (socket says 172.21.15.84:43198) 2024-04-28T12:51:27.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7af400640 1 -- 172.21.15.84:0/835216144 learned_addr learned my addr 172.21.15.84:0/835216144 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7afe00640 1 -- 172.21.15.84:0/835216144 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 msgr2=0x557317ba9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7afe00640 1 --2- 172.21.15.84:0/835216144 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317ba9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7afe00640 1 -- 172.21.15.84:0/835216144 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b9de20 msgr2=0x557317b9a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7afe00640 1 --2- 172.21.15.84:0/835216144 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b9de20 0x557317b9a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.157+0000 7ff7afe00640 1 -- 172.21.15.84:0/835216144 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557317b99540 con 0x557317b982c0 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.158+0000 7ff7afe00640 1 --2- 172.21.15.84:0/835216144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317b8edf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x557317b94bf0 tx=0x7ff7a00058b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f2ba19db652ae6d server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.158+0000 7ff79be00640 1 -- 172.21.15.84:0/835216144 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0055070 con 0x557317b982c0 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.158+0000 7ff79be00640 1 -- 172.21.15.84:0/835216144 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7a004f350 con 0x557317b982c0 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.158+0000 7ff79be00640 1 -- 172.21.15.84:0/835216144 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0037030 con 0x557317b982c0 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.159+0000 7ff7b9832740 1 -- 172.21.15.84:0/835216144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 msgr2=0x557317b8edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.159+0000 7ff7b9832740 1 --2- 172.21.15.84:0/835216144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317b8edf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x557317b94bf0 tx=0x7ff7a00058b0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.159+0000 7ff7b9832740 1 -- 172.21.15.84:0/835216144 shutdown_connections 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.159+0000 7ff7b9832740 1 --2- 172.21.15.84:0/835216144 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b9de20 0x557317b9a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.159+0000 7ff7b9832740 1 --2- 172.21.15.84:0/835216144 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317ba9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.159+0000 7ff7b9832740 1 --2- 172.21.15.84:0/835216144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b982c0 0x557317b8edf0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.159+0000 7ff7b9832740 1 -- 172.21.15.84:0/835216144 >> 172.21.15.84:0/835216144 conn(0x557317b2cd10 msgr2=0x557317ba2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:27.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.159+0000 7ff7b9832740 1 -- 172.21.15.84:0/835216144 shutdown_connections 2024-04-28T12:51:27.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.159+0000 7ff7b9832740 1 -- 172.21.15.84:0/835216144 wait complete. 2024-04-28T12:51:27.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.159+0000 7ff7b9832740 1 -- start start 2024-04-28T12:51:27.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7b9832740 1 Processor -- start 2024-04-28T12:51:27.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7b9832740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b982c0 0x557317ba5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7b9832740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317ba5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7b9832740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b9de20 0x557317ba86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7b9832740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x557317bb0620 con 0x557317b982c0 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7b9832740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557317baff50 con 0x557317b98cb0 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7b9832740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x557317bb0cf0 con 0x557317b9de20 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7b9832740 10 client.0 _ll_register_callbacks cb 0x7ff7a8099080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7af400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317ba5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7af400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317ba5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58980/0 (socket says 172.21.15.84:58980) 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7af400640 1 -- 172.21.15.84:0/1761977916 learned_addr learned my addr 172.21.15.84:0/1761977916 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7afe00640 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b982c0 0x557317ba5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.160+0000 7ff7b4a00640 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b9de20 0x557317ba86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.161+0000 7ff7af400640 1 -- 172.21.15.84:0/1761977916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b9de20 msgr2=0x557317ba86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.161+0000 7ff7af400640 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b9de20 0x557317ba86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.161+0000 7ff7af400640 1 -- 172.21.15.84:0/1761977916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b982c0 msgr2=0x557317ba5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.161+0000 7ff7af400640 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b982c0 0x557317ba5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.161+0000 7ff7af400640 1 -- 172.21.15.84:0/1761977916 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557317ba8c90 con 0x557317b98cb0 2024-04-28T12:51:27.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.161+0000 7ff7af400640 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317ba5680 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff79c003920 tx=0x7ff79c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89176ebd9e735371 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.161+0000 7ff799600640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.161+0000 7ff799600640 1 -- 172.21.15.84:0/1761977916 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c00fa80 con 0x557317b98cb0 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.161+0000 7ff7b9832740 1 -- 172.21.15.84:0/1761977916 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7a004f350 con 0x557317b98cb0 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.161+0000 7ff799600640 1 -- 172.21.15.84:0/1761977916 <== mon.2 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff79c00fc20 con 0x557317b98cb0 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.161+0000 7ff799600640 1 -- 172.21.15.84:0/1761977916 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff79c035050 con 0x557317b98cb0 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.161+0000 7ff7b9832740 10 client.25367 fetch_fsmap learned FSMap version 24 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.162+0000 7ff7b9832740 1 -- 172.21.15.84:0/1761977916 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557317c152e0 con 0x557317b98cb0 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.162+0000 7ff799600640 1 -- 172.21.15.84:0/1761977916 <== mon.2 v2:172.21.15.161:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7a004f350 con 0x557317b98cb0 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.162+0000 7ff799600640 1 -- 172.21.15.84:0/1761977916 <== mon.2 v2:172.21.15.161:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff79c020440 con 0x557317b98cb0 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.162+0000 7ff7b9832740 10 client.25367 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.162+0000 7ff7b9832740 10 client.25367 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.162+0000 7ff7b9832740 20 client.25367 populate_metadata read hostname 'smithi084' 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.162+0000 7ff7b9832740 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x557317c19e30 0x557317c18b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.162+0000 7ff7b9832740 4 client.25367 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:51:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.162+0000 7ff7b9832740 1 -- 172.21.15.84:0/1761977916 --> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] -- command(tid 0: {"prefix": "status"}) -- 0x557317b04080 con 0x557317c19e30 2024-04-28T12:51:27.162 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.162+0000 7ff7afe00640 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x557317c19e30 0x557317c18b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.162 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.163+0000 7ff7afe00640 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x557317c19e30 0x557317c18b40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=86d895e0cf4f8a41 server_cookie=3b43fbdd6b89e69e in_seq=0 out_seq=0 2024-04-28T12:51:27.162 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.163+0000 7ff799600640 10 client.25367 ms_handle_connect on v2:172.21.15.84:6838/3772865132 2024-04-28T12:51:27.162 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.163+0000 7ff799600640 1 -- 172.21.15.84:0/1761977916 <== mds.1 v2:172.21.15.84:6838/3772865132 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557317b04080 con 0x557317c19e30 2024-04-28T12:51:27.162 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.163+0000 7ff799600640 10 client.25367 handle_command_reply: tid=0 2024-04-28T12:51:27.162 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.163+0000 7ff7b9832740 1 client.25367 shutdown 2024-04-28T12:51:27.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.163+0000 7ff7b9832740 1 -- 172.21.15.84:0/1761977916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 msgr2=0x557317ba5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.163+0000 7ff7b9832740 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317ba5680 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff79c003920 tx=0x7ff79c004730 comp rx=0 tx=0).stop 2024-04-28T12:51:27.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.164+0000 7ff7b9832740 1 -- 172.21.15.84:0/1761977916 shutdown_connections 2024-04-28T12:51:27.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.164+0000 7ff7b9832740 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.84:6838/3772865132,v1:172.21.15.84:6839/3772865132] conn(0x557317c19e30 0x557317c18b40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.164+0000 7ff7b9832740 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557317b9de20 0x557317ba86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.164+0000 7ff7b9832740 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557317b98cb0 0x557317ba5680 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.164+0000 7ff799600640 0 client.25367 ms_handle_reset on v2:172.21.15.84:6838/3772865132 2024-04-28T12:51:27.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.164+0000 7ff7b9832740 1 --2- 172.21.15.84:0/1761977916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x557317b982c0 0x557317ba5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.164+0000 7ff7b9832740 1 -- 172.21.15.84:0/1761977916 >> 172.21.15.84:0/1761977916 conn(0x557317b2cd10 msgr2=0x557317b9a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:27.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.164+0000 7ff7b9832740 1 -- 172.21.15.84:0/1761977916 shutdown_connections 2024-04-28T12:51:27.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.164+0000 7ff7b9832740 1 -- 172.21.15.84:0/1761977916 wait complete. 2024-04-28T12:51:27.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.164+0000 7ff7b9832740 20 client.25367 trim_cache size 0 max 16384 2024-04-28T12:51:27.164 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-28T12:51:27.164 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_fsid": "ddee604a-055b-11ef-bc93-c7b262605968", 2024-04-28T12:51:27.164 INFO:teuthology.orchestra.run.smithi084.stdout: "whoami": 1, 2024-04-28T12:51:27.164 INFO:teuthology.orchestra.run.smithi084.stdout: "id": 24562, 2024-04-28T12:51:27.164 INFO:teuthology.orchestra.run.smithi084.stdout: "want_state": "up:active", 2024-04-28T12:51:27.164 INFO:teuthology.orchestra.run.smithi084.stdout: "state": "up:active", 2024-04-28T12:51:27.164 INFO:teuthology.orchestra.run.smithi084.stdout: "fs_name": "cephfs", 2024-04-28T12:51:27.164 INFO:teuthology.orchestra.run.smithi084.stdout: "rank_uptime": 140.44813191599999, 2024-04-28T12:51:27.164 INFO:teuthology.orchestra.run.smithi084.stdout: "mdsmap_epoch": 24, 2024-04-28T12:51:27.164 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch": 94, 2024-04-28T12:51:27.164 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch_barrier": 90, 2024-04-28T12:51:27.165 INFO:teuthology.orchestra.run.smithi084.stdout: "uptime": 185.59289867300001 2024-04-28T12:51:27.165 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-28T12:51:27.165 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.165+0000 7ff7b5400640 1 -- 172.21.15.84:0/3982405246 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7940a3c40 msgr2=0x7ff7940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.165 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.165+0000 7ff7b5400640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7940a3c40 0x7ff7940a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x557317b99510 tx=0x7ff79c013040 comp rx=0 tx=0).stop 2024-04-28T12:51:27.165 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.165+0000 7ff7b5400640 1 -- 172.21.15.84:0/3982405246 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b010a610 msgr2=0x7ff7b01a2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.165 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.165+0000 7ff7b5400640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b010a610 0x7ff7b01a2100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8009fd0 tx=0x7ff7a800ee50 comp rx=0 tx=0).stop 2024-04-28T12:51:27.165 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.165+0000 7ff7b5400640 1 -- 172.21.15.84:0/3982405246 shutdown_connections 2024-04-28T12:51:27.166 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.165+0000 7ff7b5400640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff7940a3c40 0x7ff7940a6100 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.166 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.165+0000 7ff7b5400640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b010a610 0x7ff7b01a2100 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.166 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.165+0000 7ff7b5400640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7b0076930 0x7ff7b019b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.166 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.165+0000 7ff7b5400640 1 --2- 172.21.15.84:0/3982405246 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7b0075350 0x7ff7b019aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.166 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.165+0000 7ff7b5400640 1 -- 172.21.15.84:0/3982405246 >> 172.21.15.84:0/3982405246 conn(0x7ff7b00fdcb0 msgr2=0x7ff7b0106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:27.166 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.165+0000 7ff7b5400640 1 -- 172.21.15.84:0/3982405246 shutdown_connections 2024-04-28T12:51:27.166 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.165+0000 7ff7b5400640 1 -- 172.21.15.84:0/3982405246 wait complete. 2024-04-28T12:51:27.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:26 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2195020644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:51:27.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:26 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:51:27.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:26 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3977092113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:51:27.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-28T12:51:27.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:26 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2195020644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T12:51:27.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:26 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:51:27.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:26 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3977092113' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T12:51:27.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.366+0000 7f5ea3a00640 1 -- 172.21.15.84:0/770062516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c076910 msgr2=0x7f5e9c074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.366+0000 7f5ea3a00640 1 --2- 172.21.15.84:0/770062516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c076910 0x7f5e9c074d60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5e90009900 tx=0x7f5e9002a480 comp rx=0 tx=0).stop 2024-04-28T12:51:27.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.366+0000 7f5ea3a00640 1 -- 172.21.15.84:0/770062516 shutdown_connections 2024-04-28T12:51:27.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.366+0000 7f5ea3a00640 1 --2- 172.21.15.84:0/770062516 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e9c10a800 0x7f5e9c10cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.366+0000 7f5ea3a00640 1 --2- 172.21.15.84:0/770062516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e9c075330 0x7f5e9c075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.366+0000 7f5ea3a00640 1 --2- 172.21.15.84:0/770062516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c076910 0x7f5e9c074d60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.366+0000 7f5ea3a00640 1 -- 172.21.15.84:0/770062516 >> 172.21.15.84:0/770062516 conn(0x7f5e9c0fdee0 msgr2=0x7f5e9c100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:27.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.366+0000 7f5ea3a00640 1 -- 172.21.15.84:0/770062516 shutdown_connections 2024-04-28T12:51:27.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.366+0000 7f5ea3a00640 1 -- 172.21.15.84:0/770062516 wait complete. 2024-04-28T12:51:27.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.367+0000 7f5ea3a00640 1 Processor -- start 2024-04-28T12:51:27.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.367+0000 7f5ea3a00640 1 -- start start 2024-04-28T12:51:27.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.367+0000 7f5ea3a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c075330 0x7f5e9c19adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea3a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e9c076910 0x7f5e9c19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea3a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e9c10a800 0x7f5e9c1a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea3a00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c1106c0 con 0x7f5e9c076910 2024-04-28T12:51:27.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea3a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c110540 con 0x7f5e9c075330 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea3a00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c110840 con 0x7f5e9c10a800 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea1000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c075330 0x7f5e9c19adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea1000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c075330 0x7f5e9c19adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58984/0 (socket says 172.21.15.84:58984) 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea1000640 1 -- 172.21.15.84:0/4153283673 learned_addr learned my addr 172.21.15.84:0/4153283673 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5e9be00640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e9c076910 0x7f5e9c19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea1a00640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e9c10a800 0x7f5e9c1a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea1000640 1 -- 172.21.15.84:0/4153283673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e9c10a800 msgr2=0x7f5e9c1a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea1000640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e9c10a800 0x7f5e9c1a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea1000640 1 -- 172.21.15.84:0/4153283673 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e9c076910 msgr2=0x7f5e9c19b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea1000640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e9c076910 0x7f5e9c19b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.368+0000 7f5ea1000640 1 -- 172.21.15.84:0/4153283673 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9c1a29c0 con 0x7f5e9c075330 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.369+0000 7f5ea1000640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c075330 0x7f5e9c19adb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5e90005ab0 tx=0x7f5e90003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.369+0000 7f5e99600640 1 -- 172.21.15.84:0/4153283673 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9003f070 con 0x7f5e9c075330 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.369+0000 7f5e99600640 1 -- 172.21.15.84:0/4153283673 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e90031c50 con 0x7f5e9c075330 2024-04-28T12:51:27.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.369+0000 7f5e99600640 1 -- 172.21.15.84:0/4153283673 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e90035030 con 0x7f5e9c075330 2024-04-28T12:51:27.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.369+0000 7f5ea3a00640 1 -- 172.21.15.84:0/4153283673 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9c1a2c50 con 0x7f5e9c075330 2024-04-28T12:51:27.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.369+0000 7f5ea3a00640 1 -- 172.21.15.84:0/4153283673 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e9c1a30f0 con 0x7f5e9c075330 2024-04-28T12:51:27.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.373+0000 7f5ea6855740 1 Processor -- start 2024-04-28T12:51:27.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5ea6855740 1 -- start start 2024-04-28T12:51:27.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5e99600640 1 -- 172.21.15.84:0/4153283673 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5e90042070 con 0x7f5e9c075330 2024-04-28T12:51:27.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5ea6855740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb555f5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5ea6855740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb555fa5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5ea6855740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb555fd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5ea6855740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x55fb55559600 con 0x55fb555f5ae0 2024-04-28T12:51:27.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5ea6855740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55fb554f6100 con 0x55fb555fac50 2024-04-28T12:51:27.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5ea6855740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55fb555555c0 con 0x55fb555f50f0 2024-04-28T12:51:27.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5e9be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb555fd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5ea1000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb555fa5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5e9be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb555fd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:58994/0 (socket says 172.21.15.84:58994) 2024-04-28T12:51:27.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5e9be00640 1 -- 172.21.15.84:0/2934920329 learned_addr learned my addr 172.21.15.84:0/2934920329 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:27.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.374+0000 7f5ea1a00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb555f5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.375+0000 7f5e9be00640 1 -- 172.21.15.84:0/2934920329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 msgr2=0x55fb555f5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.375+0000 7f5e9be00640 1 --2- 172.21.15.84:0/2934920329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb555f5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.375+0000 7f5e9be00640 1 -- 172.21.15.84:0/2934920329 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 msgr2=0x55fb555fa5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.375+0000 7f5e9be00640 1 --2- 172.21.15.84:0/2934920329 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb555fa5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.375+0000 7f5e9be00640 1 -- 172.21.15.84:0/2934920329 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb5526b2d0 con 0x55fb555fac50 2024-04-28T12:51:27.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.375+0000 7f5e9be00640 1 --2- 172.21.15.84:0/2934920329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb555fd0e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009cf0 tx=0x7f5e88011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bb477f23b8ddccc server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:27.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.375+0000 7f5e8e000640 1 -- 172.21.15.84:0/2934920329 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8801e070 con 0x55fb555fac50 2024-04-28T12:51:27.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.375+0000 7f5e8e000640 1 -- 172.21.15.84:0/2934920329 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e88019ae0 con 0x55fb555fac50 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.375+0000 7f5e8e000640 1 -- 172.21.15.84:0/2934920329 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8800e9e0 con 0x55fb555fac50 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.376+0000 7f5e99600640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5e840a3cc0 0x7f5e840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.376+0000 7f5e99600640 1 -- 172.21.15.84:0/4153283673 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f5e900f1000 con 0x7f5e9c075330 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.376+0000 7f5ea1a00640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5e840a3cc0 0x7f5e840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.376+0000 7f5ea6855740 1 -- 172.21.15.84:0/2934920329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 msgr2=0x55fb555fd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.376+0000 7f5ea6855740 1 --2- 172.21.15.84:0/2934920329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb555fd0e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009cf0 tx=0x7f5e88011810 comp rx=0 tx=0).stop 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.377+0000 7f5ea6855740 1 -- 172.21.15.84:0/2934920329 shutdown_connections 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.377+0000 7f5ea6855740 1 --2- 172.21.15.84:0/2934920329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb555fd0e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.377+0000 7f5ea6855740 1 --2- 172.21.15.84:0/2934920329 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb555fa5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.377+0000 7f5ea6855740 1 --2- 172.21.15.84:0/2934920329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb555f5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.377+0000 7f5ea6855740 1 -- 172.21.15.84:0/2934920329 >> 172.21.15.84:0/2934920329 conn(0x55fb55589d10 msgr2=0x55fb555f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.377+0000 7f5ea6855740 1 -- 172.21.15.84:0/2934920329 shutdown_connections 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.377+0000 7f5ea1a00640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5e840a3cc0 0x7f5e840a6180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55fb555f6340 tx=0x7f5e94009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:27.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.377+0000 7f5ea6855740 1 -- 172.21.15.84:0/2934920329 wait complete. 2024-04-28T12:51:27.377 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.377+0000 7f5ea6855740 1 -- start start 2024-04-28T12:51:27.377 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.377+0000 7f5ea6855740 1 Processor -- start 2024-04-28T12:51:27.377 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea6855740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb555ff580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.377 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea6855740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555f5ae0 0x55fb555ffac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea6855740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555fac50 0x55fb55606b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea6855740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x55fb555fdc00 con 0x55fb555fac50 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea6855740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55fb554e0f70 con 0x55fb555f5ae0 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea6855740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55fb555fdd80 con 0x55fb555f50f0 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea6855740 10 client.0 _ll_register_callbacks cb 0x55fb5554f480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea1000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb555ff580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea1a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555fac50 0x55fb55606b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea1000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb555ff580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:44374/0 (socket says 172.21.15.84:44374) 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea1000640 1 -- 172.21.15.84:0/3219377307 learned_addr learned my addr 172.21.15.84:0/3219377307 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5e9be00640 1 --2- 172.21.15.84:0/3219377307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555f5ae0 0x55fb555ffac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea1000640 1 -- 172.21.15.84:0/3219377307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555f5ae0 msgr2=0x55fb555ffac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea1000640 1 --2- 172.21.15.84:0/3219377307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555f5ae0 0x55fb555ffac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea1000640 1 -- 172.21.15.84:0/3219377307 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555fac50 msgr2=0x55fb55606b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.378 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea1000640 1 --2- 172.21.15.84:0/3219377307 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555fac50 0x55fb55606b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.378+0000 7f5ea1000640 1 -- 172.21.15.84:0/3219377307 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb553356b0 con 0x55fb555f50f0 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.379+0000 7f5ea1000640 1 --2- 172.21.15.84:0/3219377307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb555ff580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5e900ee660 tx=0x7f5e90003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d69fdf55d99c1f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.379+0000 7f5e82a00640 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.379+0000 7f5e82a00640 1 -- 172.21.15.84:0/3219377307 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e90049050 con 0x55fb555f50f0 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.379+0000 7f5e82a00640 1 -- 172.21.15.84:0/3219377307 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f5e9003b460 con 0x55fb555f50f0 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.379+0000 7f5ea6855740 1 -- 172.21.15.84:0/3219377307 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e88019ae0 con 0x55fb555f50f0 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.379+0000 7f5e82a00640 1 -- 172.21.15.84:0/3219377307 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e900bcc80 con 0x55fb555f50f0 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.379+0000 7f5ea6855740 10 client.25366 fetch_fsmap learned FSMap version 24 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.379+0000 7f5ea6855740 1 -- 172.21.15.84:0/3219377307 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fb555fdc70 con 0x55fb555f50f0 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.379+0000 7f5e82a00640 1 -- 172.21.15.84:0/3219377307 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e88019ae0 con 0x55fb555f50f0 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.380+0000 7f5e82a00640 1 -- 172.21.15.84:0/3219377307 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5e900b9490 con 0x55fb555f50f0 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.380+0000 7f5ea6855740 10 client.25366 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.380+0000 7f5ea6855740 10 client.25366 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.380+0000 7f5ea6855740 20 client.25366 populate_metadata read hostname 'smithi084' 2024-04-28T12:51:27.379 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.380+0000 7f5ea6855740 1 --2- 172.21.15.84:0/3219377307 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55fb5560c1c0 0x55fb5560d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.380 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.380+0000 7f5ea6855740 4 client.25366 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-28T12:51:27.380 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.380+0000 7f5ea6855740 1 -- 172.21.15.84:0/3219377307 --> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb55567210 con 0x55fb5560c1c0 2024-04-28T12:51:27.380 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.380+0000 7f5e9be00640 1 --2- 172.21.15.84:0/3219377307 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55fb5560c1c0 0x55fb5560d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.380 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.381+0000 7f5e9be00640 1 --2- 172.21.15.84:0/3219377307 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55fb5560c1c0 0x55fb5560d630 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6a0f3b80740534c server_cookie=187537850d4b762f in_seq=0 out_seq=0 2024-04-28T12:51:27.380 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.381+0000 7f5e82a00640 10 client.25366 ms_handle_connect on v2:172.21.15.161:6838/3812720642 2024-04-28T12:51:27.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.381+0000 7f5e82a00640 1 -- 172.21.15.84:0/3219377307 <== mds.0 v2:172.21.15.161:6838/3812720642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fb55567210 con 0x55fb5560c1c0 2024-04-28T12:51:27.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.381+0000 7f5e82a00640 10 client.25366 handle_command_reply: tid=0 2024-04-28T12:51:27.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5ea6855740 1 client.25366 shutdown 2024-04-28T12:51:27.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5ea6855740 1 -- 172.21.15.84:0/3219377307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 msgr2=0x55fb555ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5ea6855740 1 --2- 172.21.15.84:0/3219377307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb555ff580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5e900ee660 tx=0x7f5e90003f90 comp rx=0 tx=0).stop 2024-04-28T12:51:27.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5ea6855740 1 -- 172.21.15.84:0/3219377307 shutdown_connections 2024-04-28T12:51:27.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5ea6855740 1 --2- 172.21.15.84:0/3219377307 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55fb5560c1c0 0x55fb5560d630 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5ea6855740 1 --2- 172.21.15.84:0/3219377307 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555fac50 0x55fb55606b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5e82a00640 0 client.25366 ms_handle_reset on v2:172.21.15.161:6838/3812720642 2024-04-28T12:51:27.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5ea6855740 1 --2- 172.21.15.84:0/3219377307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555f5ae0 0x55fb555ffac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.381 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5ea6855740 1 --2- 172.21.15.84:0/3219377307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb555ff580 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.382 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5ea6855740 1 -- 172.21.15.84:0/3219377307 >> 172.21.15.84:0/3219377307 conn(0x55fb55589d10 msgr2=0x55fb555efeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:27.382 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5ea6855740 1 -- 172.21.15.84:0/3219377307 shutdown_connections 2024-04-28T12:51:27.382 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5ea6855740 1 -- 172.21.15.84:0/3219377307 wait complete. 2024-04-28T12:51:27.382 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.382+0000 7f5ea6855740 20 client.25366 trim_cache size 0 max 16384 2024-04-28T12:51:27.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.423+0000 7f5ea6855740 1 Processor -- start 2024-04-28T12:51:27.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.424+0000 7f5ea6855740 1 -- start start 2024-04-28T12:51:27.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.424+0000 7f5ea6855740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb5560b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.424+0000 7f5ea6855740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb55606ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.424+0000 7f5ea6855740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb555ff190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.424+0000 7f5ea6855740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x55fb5560b970 con 0x55fb555f5ae0 2024-04-28T12:51:27.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.424+0000 7f5ea6855740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55fb5560c040 con 0x55fb555fac50 2024-04-28T12:51:27.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.424+0000 7f5ea6855740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55fb5560c710 con 0x55fb555f50f0 2024-04-28T12:51:27.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.424+0000 7f5ea1a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb555ff190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.424+0000 7f5e9be00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb55606ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.424+0000 7f5ea1a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb555ff190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:59024/0 (socket says 172.21.15.84:59024) 2024-04-28T12:51:27.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.424+0000 7f5ea1a00640 1 -- 172.21.15.84:0/2978548475 learned_addr learned my addr 172.21.15.84:0/2978548475 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:27.422 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.424+0000 7f5ea1000640 1 --2- 172.21.15.84:0/2978548475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb5560b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.425+0000 7f5ea1000640 1 -- 172.21.15.84:0/2978548475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 msgr2=0x55fb555ff190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.425+0000 7f5ea1000640 1 --2- 172.21.15.84:0/2978548475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb555ff190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.425+0000 7f5ea1000640 1 -- 172.21.15.84:0/2978548475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 msgr2=0x55fb55606ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.425+0000 7f5ea1000640 1 --2- 172.21.15.84:0/2978548475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb55606ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.425+0000 7f5ea1000640 1 -- 172.21.15.84:0/2978548475 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb555e2400 con 0x55fb555f50f0 2024-04-28T12:51:27.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.425+0000 7f5ea1000640 1 --2- 172.21.15.84:0/2978548475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb5560b420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5e900bce40 tx=0x7f5e900f4bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28d8a548ab348da7 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:27.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.425+0000 7f5e8d600640 1 -- 172.21.15.84:0/2978548475 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e900d5030 con 0x55fb555f50f0 2024-04-28T12:51:27.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.425+0000 7f5e8d600640 1 -- 172.21.15.84:0/2978548475 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e900bceb0 con 0x55fb555f50f0 2024-04-28T12:51:27.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.425+0000 7f5e8d600640 1 -- 172.21.15.84:0/2978548475 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e90049070 con 0x55fb555f50f0 2024-04-28T12:51:27.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.426+0000 7f5ea6855740 1 -- 172.21.15.84:0/2978548475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 msgr2=0x55fb5560b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.426+0000 7f5ea6855740 1 --2- 172.21.15.84:0/2978548475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb5560b420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5e900bce40 tx=0x7f5e900f4bc0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.427+0000 7f5ea6855740 1 -- 172.21.15.84:0/2978548475 shutdown_connections 2024-04-28T12:51:27.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.427+0000 7f5ea6855740 1 --2- 172.21.15.84:0/2978548475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb555ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.427+0000 7f5ea6855740 1 --2- 172.21.15.84:0/2978548475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb55606ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.427+0000 7f5ea6855740 1 --2- 172.21.15.84:0/2978548475 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb5560b420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.427+0000 7f5ea6855740 1 -- 172.21.15.84:0/2978548475 >> 172.21.15.84:0/2978548475 conn(0x55fb55589d10 msgr2=0x55fb5560ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:27.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.427+0000 7f5ea6855740 1 -- 172.21.15.84:0/2978548475 shutdown_connections 2024-04-28T12:51:27.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.427+0000 7f5ea6855740 1 -- 172.21.15.84:0/2978548475 wait complete. 2024-04-28T12:51:27.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.427+0000 7f5ea6855740 1 -- start start 2024-04-28T12:51:27.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.428+0000 7f5ea6855740 1 Processor -- start 2024-04-28T12:51:27.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.428+0000 7f5ea6855740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb55602710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.428+0000 7f5ea6855740 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb55602c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.428+0000 7f5ea6855740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb55605c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.428+0000 7f5ea6855740 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x55fb5560d4b0 con 0x55fb555f5ae0 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.428+0000 7f5ea6855740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55fb5560cde0 con 0x55fb555fac50 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.428+0000 7f5ea6855740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x55fb5560db80 con 0x55fb555f50f0 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.428+0000 7f5ea6855740 10 client.0 _ll_register_callbacks cb 0x7f5e900f4840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.428+0000 7f5e9be00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb55602c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.429+0000 7f5ea1a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb55605c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.429+0000 7f5ea1000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb55602710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.429+0000 7f5e9be00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb55602c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43254/0 (socket says 172.21.15.84:43254) 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.429+0000 7f5e9be00640 1 -- 172.21.15.84:0/1844868640 learned_addr learned my addr 172.21.15.84:0/1844868640 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.429+0000 7f5e9be00640 1 -- 172.21.15.84:0/1844868640 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 msgr2=0x55fb55602710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.429+0000 7f5e9be00640 1 --2- 172.21.15.84:0/1844868640 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb55602710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.429+0000 7f5e9be00640 1 -- 172.21.15.84:0/1844868640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 msgr2=0x55fb55605c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.429+0000 7f5e9be00640 1 --2- 172.21.15.84:0/1844868640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb55605c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.429+0000 7f5e9be00640 1 -- 172.21.15.84:0/1844868640 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb55606260 con 0x55fb555f5ae0 2024-04-28T12:51:27.428 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.429+0000 7f5ea1a00640 1 --2- 172.21.15.84:0/1844868640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb55605c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.429+0000 7f5e9be00640 1 --2- 172.21.15.84:0/1844868640 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb55602c50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5e880058c0 tx=0x7f5e88005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d84050716beb7c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.429+0000 7f5e82a00640 10 client.0 ms_handle_connect on v2:172.21.15.84:3300/0 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.430+0000 7f5e82a00640 1 -- 172.21.15.84:0/1844868640 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8801e040 con 0x55fb555f5ae0 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.430+0000 7f5e82a00640 1 -- 172.21.15.84:0/1844868640 <== mon.0 v2:172.21.15.84:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f5e880197d0 con 0x55fb555f5ae0 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.430+0000 7f5ea6855740 1 -- 172.21.15.84:0/1844868640 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e900bceb0 con 0x55fb555f5ae0 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.430+0000 7f5e82a00640 1 -- 172.21.15.84:0/1844868640 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5e8803b050 con 0x55fb555f5ae0 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.430+0000 7f5ea6855740 10 client.15264 fetch_fsmap learned FSMap version 24 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.430+0000 7f5ea6855740 1 -- 172.21.15.84:0/1844868640 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fb556726e0 con 0x55fb555f5ae0 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.430+0000 7f5e82a00640 1 -- 172.21.15.84:0/1844868640 <== mon.0 v2:172.21.15.84:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e900bceb0 con 0x55fb555f5ae0 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.430+0000 7f5e82a00640 1 -- 172.21.15.84:0/1844868640 <== mon.0 v2:172.21.15.84:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f5e8802b370 con 0x55fb555f5ae0 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.431+0000 7f5ea6855740 10 client.15264 fetch_fsmap finished waiting for FSMap version 24 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.431+0000 7f5ea6855740 10 client.15264 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.431+0000 7f5ea6855740 20 client.15264 populate_metadata read hostname 'smithi084' 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.431+0000 7f5ea6855740 1 --2- 172.21.15.84:0/1844868640 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55fb55677030 0x55fb55675d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:27.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.431+0000 7f5ea6855740 4 client.15264 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-28T12:51:27.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.431+0000 7f5ea6855740 1 -- 172.21.15.84:0/1844868640 --> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] -- command(tid 0: {"prefix": "status"}) -- 0x55fb55561080 con 0x55fb55677030 2024-04-28T12:51:27.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.431+0000 7f5ea1000640 1 --2- 172.21.15.84:0/1844868640 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55fb55677030 0x55fb55675d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:27.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea1000640 1 --2- 172.21.15.84:0/1844868640 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55fb55677030 0x55fb55675d40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98ac4c87ccfa1f76 server_cookie=8f9344655c9414a8 in_seq=0 out_seq=0 2024-04-28T12:51:27.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5e82a00640 10 client.15264 ms_handle_connect on v2:172.21.15.161:6838/3812720642 2024-04-28T12:51:27.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5e82a00640 1 -- 172.21.15.84:0/1844868640 <== mds.0 v2:172.21.15.161:6838/3812720642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fb55561080 con 0x55fb55677030 2024-04-28T12:51:27.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5e82a00640 10 client.15264 handle_command_reply: tid=0 2024-04-28T12:51:27.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea6855740 1 client.15264 shutdown 2024-04-28T12:51:27.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea6855740 1 -- 172.21.15.84:0/1844868640 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 msgr2=0x55fb55602c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea6855740 1 --2- 172.21.15.84:0/1844868640 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb55602c50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5e880058c0 tx=0x7f5e88005f10 comp rx=0 tx=0).stop 2024-04-28T12:51:27.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea6855740 1 -- 172.21.15.84:0/1844868640 shutdown_connections 2024-04-28T12:51:27.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea6855740 1 --2- 172.21.15.84:0/1844868640 >> [v2:172.21.15.161:6838/3812720642,v1:172.21.15.161:6839/3812720642] conn(0x55fb55677030 0x55fb55675d40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea6855740 1 --2- 172.21.15.84:0/1844868640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fb555fac50 0x55fb55605c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5e82a00640 0 client.15264 ms_handle_reset on v2:172.21.15.161:6838/3812720642 2024-04-28T12:51:27.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea6855740 1 --2- 172.21.15.84:0/1844868640 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x55fb555f5ae0 0x55fb55602c50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea6855740 1 --2- 172.21.15.84:0/1844868640 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb555f50f0 0x55fb55602710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea6855740 1 -- 172.21.15.84:0/1844868640 >> 172.21.15.84:0/1844868640 conn(0x55fb55589d10 msgr2=0x55fb555ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:27.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea6855740 1 -- 172.21.15.84:0/1844868640 shutdown_connections 2024-04-28T12:51:27.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea6855740 1 -- 172.21.15.84:0/1844868640 wait complete. 2024-04-28T12:51:27.432 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.432+0000 7f5ea6855740 20 client.15264 trim_cache size 0 max 16384 2024-04-28T12:51:27.432 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-04-28T12:51:27.432 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_fsid": "ddee604a-055b-11ef-bc93-c7b262605968", 2024-04-28T12:51:27.432 INFO:teuthology.orchestra.run.smithi084.stdout: "whoami": 0, 2024-04-28T12:51:27.433 INFO:teuthology.orchestra.run.smithi084.stdout: "id": 24602, 2024-04-28T12:51:27.433 INFO:teuthology.orchestra.run.smithi084.stdout: "want_state": "up:active", 2024-04-28T12:51:27.433 INFO:teuthology.orchestra.run.smithi084.stdout: "state": "up:active", 2024-04-28T12:51:27.433 INFO:teuthology.orchestra.run.smithi084.stdout: "fs_name": "cephfs", 2024-04-28T12:51:27.433 INFO:teuthology.orchestra.run.smithi084.stdout: "rank_uptime": 172.25378696499999, 2024-04-28T12:51:27.433 INFO:teuthology.orchestra.run.smithi084.stdout: "mdsmap_epoch": 24, 2024-04-28T12:51:27.433 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch": 94, 2024-04-28T12:51:27.433 INFO:teuthology.orchestra.run.smithi084.stdout: "osdmap_epoch_barrier": 90, 2024-04-28T12:51:27.433 INFO:teuthology.orchestra.run.smithi084.stdout: "uptime": 178.96047317200001 2024-04-28T12:51:27.433 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-04-28T12:51:27.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.433+0000 7f5ea3a00640 1 -- 172.21.15.84:0/4153283673 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5e840a3cc0 msgr2=0x7f5e840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.433+0000 7f5ea3a00640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5e840a3cc0 0x7f5e840a6180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55fb555f6340 tx=0x7f5e94009500 comp rx=0 tx=0).stop 2024-04-28T12:51:27.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.433+0000 7f5ea3a00640 1 -- 172.21.15.84:0/4153283673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c075330 msgr2=0x7f5e9c19adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:27.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.433+0000 7f5ea3a00640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c075330 0x7f5e9c19adb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5e90005ab0 tx=0x7f5e90003b90 comp rx=0 tx=0).stop 2024-04-28T12:51:27.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.433+0000 7f5ea3a00640 1 -- 172.21.15.84:0/4153283673 shutdown_connections 2024-04-28T12:51:27.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.433+0000 7f5ea3a00640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f5e840a3cc0 0x7f5e840a6180 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.433+0000 7f5ea3a00640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e9c10a800 0x7f5e9c1a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.433+0000 7f5ea3a00640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e9c076910 0x7f5e9c19b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.433+0000 7f5ea3a00640 1 --2- 172.21.15.84:0/4153283673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e9c075330 0x7f5e9c19adb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:27.434 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.433+0000 7f5ea3a00640 1 -- 172.21.15.84:0/4153283673 >> 172.21.15.84:0/4153283673 conn(0x7f5e9c0fdee0 msgr2=0x7f5e9c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:27.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.433+0000 7f5ea3a00640 1 -- 172.21.15.84:0/4153283673 shutdown_connections 2024-04-28T12:51:27.435 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:27.433+0000 7f5ea3a00640 1 -- 172.21.15.84:0/4153283673 wait complete. 2024-04-28T12:51:27.454 INFO:teuthology.run_tasks:Running task sequential... 2024-04-28T12:51:27.464 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-28T12:51:27.465 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-28T12:51:27.465 INFO:teuthology.task.print:Enabling mgr modules 2024-04-28T12:51:27.465 INFO:teuthology.run_tasks:Running task kclient... 2024-04-28T12:51:27.488 INFO:tasks.kclient:Mounting kernel clients... 2024-04-28T12:51:27.488 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-28T12:51:27.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:51:27.489 DEBUG:teuthology.orchestra.run.smithi084:> ip netns list 2024-04-28T12:51:27.540 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-28T12:51:27.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:51:27.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-28T12:51:27.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:51:27.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip link delete ceph-brx 2024-04-28T12:51:27.716 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-28T12:51:27.716 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-28T12:51:27.717 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-28T12:51:27.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid ddee604a-055b-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-28T12:51:27.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:27 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:27 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:27 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:27 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:27 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:27 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:27 smithi161 ceph-mon[62387]: pgmap v413: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:27.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:27 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:27 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:27 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:27 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:27 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:27 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:27 smithi179 ceph-mon[63029]: pgmap v413: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:27.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:27 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:27 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:27 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:27 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:27 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:27 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:27.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:27 smithi084 ceph-mon[28081]: pgmap v413: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:28.440 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/mon.a/config 2024-04-28T12:51:29.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:29.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:29.744 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:29.744 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:29 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:29.844 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:29.844 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:29.844 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:29.844 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:29 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:29.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:29.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:29.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:29.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:29 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:30.636 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:30 smithi084 ceph-mon[28081]: pgmap v414: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:30.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:30 smithi179 ceph-mon[63029]: pgmap v414: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:30.993 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:30 smithi161 ceph-mon[62387]: pgmap v414: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:51:31.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.157+0000 7f9270e00700 1 -- 172.21.15.84:0/4273996917 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f926c075f70 msgr2=0x7f926c10f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:31.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.157+0000 7f9270e00700 1 --2- 172.21.15.84:0/4273996917 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f926c075f70 0x7f926c10f8e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9258009990 tx=0x7f9258009ca0 comp rx=0 tx=0).stop 2024-04-28T12:51:31.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.157+0000 7f9270e00700 1 -- 172.21.15.84:0/4273996917 shutdown_connections 2024-04-28T12:51:31.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.157+0000 7f9270e00700 1 --2- 172.21.15.84:0/4273996917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c10fe20 0x7f926c112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:31.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.157+0000 7f9270e00700 1 --2- 172.21.15.84:0/4273996917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f926c075650 0x7f926c075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:31.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.157+0000 7f9270e00700 1 --2- 172.21.15.84:0/4273996917 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f926c075f70 0x7f926c10f8e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:31.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.157+0000 7f9270e00700 1 -- 172.21.15.84:0/4273996917 >> 172.21.15.84:0/4273996917 conn(0x7f926c07a580 msgr2=0x7f926c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:31.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.157+0000 7f9270e00700 1 -- 172.21.15.84:0/4273996917 shutdown_connections 2024-04-28T12:51:31.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.157+0000 7f9270e00700 1 -- 172.21.15.84:0/4273996917 wait complete. 2024-04-28T12:51:31.155 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.158+0000 7f9270e00700 1 Processor -- start 2024-04-28T12:51:31.155 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.158+0000 7f9270e00700 1 -- start start 2024-04-28T12:51:31.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.159+0000 7f9270e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f926c075650 0x7f926c19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:31.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.159+0000 7f9270e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c075f70 0x7f926c19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:31.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.159+0000 7f9270e00700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f926c10fe20 0x7f926c1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:31.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.159+0000 7f9270e00700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f926c115840 con 0x7f926c075650 2024-04-28T12:51:31.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.159+0000 7f9270e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1156c0 con 0x7f926c075f70 2024-04-28T12:51:31.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.159+0000 7f9270e00700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1159c0 con 0x7f926c10fe20 2024-04-28T12:51:31.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.159+0000 7f9269e00700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f926c075650 0x7f926c19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:31.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.159+0000 7f9269400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c075f70 0x7f926c19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:31.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.159+0000 7f9269400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c075f70 0x7f926c19c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:39872/0 (socket says 172.21.15.84:39872) 2024-04-28T12:51:31.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.159+0000 7f9269400700 1 -- 172.21.15.84:0/1042648654 learned_addr learned my addr 172.21.15.84:0/1042648654 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:31.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.159+0000 7f926a800700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f926c10fe20 0x7f926c1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:31.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.160+0000 7f9269400700 1 -- 172.21.15.84:0/1042648654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f926c10fe20 msgr2=0x7f926c1a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:31.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.160+0000 7f9269400700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f926c10fe20 0x7f926c1a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:31.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.160+0000 7f9269400700 1 -- 172.21.15.84:0/1042648654 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f926c075650 msgr2=0x7f926c19bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:31.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.160+0000 7f9269400700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f926c075650 0x7f926c19bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:31.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.160+0000 7f9269400700 1 -- 172.21.15.84:0/1042648654 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c1a0d50 con 0x7f926c075f70 2024-04-28T12:51:31.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.160+0000 7f9269400700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c075f70 0x7f926c19c2d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9258000c00 tx=0x7f92580057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:31.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.160+0000 7f9262a00700 1 -- 172.21.15.84:0/1042648654 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258027070 con 0x7f926c075f70 2024-04-28T12:51:31.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.160+0000 7f9270e00700 1 -- 172.21.15.84:0/1042648654 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926c1a0fe0 con 0x7f926c075f70 2024-04-28T12:51:31.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.160+0000 7f9262a00700 1 -- 172.21.15.84:0/1042648654 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9258022470 con 0x7f926c075f70 2024-04-28T12:51:31.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.160+0000 7f9262a00700 1 -- 172.21.15.84:0/1042648654 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258017650 con 0x7f926c075f70 2024-04-28T12:51:31.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.161+0000 7f9270e00700 1 -- 172.21.15.84:0/1042648654 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f926c1a88c0 con 0x7f926c075f70 2024-04-28T12:51:31.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.162+0000 7f9270e00700 1 -- 172.21.15.84:0/1042648654 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f926c076d40 con 0x7f926c075f70 2024-04-28T12:51:31.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.164+0000 7f9262a00700 1 -- 172.21.15.84:0/1042648654 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f925802a070 con 0x7f926c075f70 2024-04-28T12:51:31.162 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.165+0000 7f9262a00700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f92500a3cd0 0x7f92500a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:31.162 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.166+0000 7f9262a00700 1 -- 172.21.15.84:0/1042648654 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f92580d15e0 con 0x7f926c075f70 2024-04-28T12:51:31.162 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.166+0000 7f9269e00700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f92500a3cd0 0x7f92500a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:31.163 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.166+0000 7f9269e00700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f92500a3cd0 0x7f92500a6190 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f925c009c60 tx=0x7f925c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:31.164 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.168+0000 7f9262a00700 1 -- 172.21.15.84:0/1042648654 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9258013070 con 0x7f926c075f70 2024-04-28T12:51:31.470 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.473+0000 7f9270e00700 1 -- 172.21.15.84:0/1042648654 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f926c02e490 con 0x7f926c075f70 2024-04-28T12:51:31.470 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.474+0000 7f9262a00700 1 -- 172.21.15.84:0/1042648654 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7f925809db10 con 0x7f926c075f70 2024-04-28T12:51:31.470 INFO:teuthology.orchestra.run.smithi084.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-28T12:51:31.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.476+0000 7f9270e00700 1 -- 172.21.15.84:0/1042648654 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f92500a3cd0 msgr2=0x7f92500a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:31.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.476+0000 7f9270e00700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f92500a3cd0 0x7f92500a6190 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f925c009c60 tx=0x7f925c009260 comp rx=0 tx=0).stop 2024-04-28T12:51:31.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.476+0000 7f9270e00700 1 -- 172.21.15.84:0/1042648654 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c075f70 msgr2=0x7f926c19c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:31.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.476+0000 7f9270e00700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c075f70 0x7f926c19c2d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9258000c00 tx=0x7f92580057c0 comp rx=0 tx=0).stop 2024-04-28T12:51:31.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.477+0000 7f9270e00700 1 -- 172.21.15.84:0/1042648654 shutdown_connections 2024-04-28T12:51:31.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.477+0000 7f9270e00700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f926c10fe20 0x7f926c1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:31.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.477+0000 7f9270e00700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f926c075650 0x7f926c19bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:31.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.477+0000 7f9270e00700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f92500a3cd0 0x7f92500a6190 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:31.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.477+0000 7f9270e00700 1 --2- 172.21.15.84:0/1042648654 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f926c075f70 0x7f926c19c2d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:31.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.477+0000 7f9270e00700 1 -- 172.21.15.84:0/1042648654 >> 172.21.15.84:0/1042648654 conn(0x7f926c07a580 msgr2=0x7f926c1006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:31.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.477+0000 7f9270e00700 1 -- 172.21.15.84:0/1042648654 shutdown_connections 2024-04-28T12:51:31.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:31.477+0000 7f9270e00700 1 -- 172.21.15.84:0/1042648654 wait complete. 2024-04-28T12:51:32.356 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-28T12:51:32.356 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-28T12:51:32.357 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi084.front.sepia.ceph.com 2024-04-28T12:51:32.357 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-28T12:51:32.357 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-28T12:51:32.357 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-28T12:51:32.357 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-28T12:51:32.357 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-28T12:51:32.357 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-28T12:51:32.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:51:32.357 DEBUG:teuthology.orchestra.run.smithi084:> ip addr 2024-04-28T12:51:32.404 INFO:teuthology.orchestra.run.smithi084.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-28T12:51:32.404 INFO:teuthology.orchestra.run.smithi084.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-28T12:51:32.404 INFO:teuthology.orchestra.run.smithi084.stdout: inet 127.0.0.1/8 scope host lo 2024-04-28T12:51:32.404 INFO:teuthology.orchestra.run.smithi084.stdout: valid_lft forever preferred_lft forever 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout: inet6 ::1/128 scope host 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout: valid_lft forever preferred_lft forever 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout: link/ether 0c:c4:7a:d8:da:68 brd ff:ff:ff:ff:ff:ff 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout: altname enp7s0f0 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout: link/ether 0c:c4:7a:d8:da:69 brd ff:ff:ff:ff:ff:ff 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout: altname enp7s0f1 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout: link/ether 0c:c4:7a:8f:d7:50 brd ff:ff:ff:ff:ff:ff 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout: link/ether 0c:c4:7a:8f:d7:51 brd ff:ff:ff:ff:ff:ff 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout: inet 172.21.15.84/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout: valid_lft 41677sec preferred_lft 41677sec 2024-04-28T12:51:32.405 INFO:teuthology.orchestra.run.smithi084.stdout: inet6 fe80::ec4:7aff:fe8f:d751/64 scope link 2024-04-28T12:51:32.406 INFO:teuthology.orchestra.run.smithi084.stdout: valid_lft forever preferred_lft forever 2024-04-28T12:51:32.406 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-28T12:51:32.406 DEBUG:teuthology.orchestra.run.smithi084:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:51:32.406 DEBUG:teuthology.orchestra.run.smithi084:> set -e 2024-04-28T12:51:32.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip link add name ceph-brx type bridge 2024-04-28T12:51:32.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip addr flush dev ceph-brx 2024-04-28T12:51:32.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip link set ceph-brx up 2024-04-28T12:51:32.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-28T12:51:32.406 DEBUG:teuthology.orchestra.run.smithi084:> ') 2024-04-28T12:51:32.533 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:51:32.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:32.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:32 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:32.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:32 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1042648654' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T12:51:32.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:32 smithi084 ceph-mon[28081]: pgmap v415: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-28T12:51:32.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:51:32.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:51:32.666 DEBUG:teuthology.orchestra.run.smithi084:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-28T12:51:32.667 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:32.667 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:32 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:32.667 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:32 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1042648654' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T12:51:32.667 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:32 smithi179 ceph-mon[63029]: pgmap v415: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-28T12:51:32.726 INFO:teuthology.orchestra.run.smithi084.stdout:1 2024-04-28T12:51:32.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:51:32.730 DEBUG:teuthology.orchestra.run.smithi084:> route 2024-04-28T12:51:32.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:32.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:32 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:32.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:32 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1042648654' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-28T12:51:32.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:32 smithi161 ceph-mon[62387]: pgmap v415: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-28T12:51:32.768 INFO:teuthology.orchestra.run.smithi084.stdout:Kernel IP routing table 2024-04-28T12:51:32.768 INFO:teuthology.orchestra.run.smithi084.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-28T12:51:32.769 INFO:teuthology.orchestra.run.smithi084.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-28T12:51:32.769 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-28T12:51:32.769 INFO:teuthology.orchestra.run.smithi084.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-28T12:51:32.769 DEBUG:teuthology.orchestra.run.smithi084:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:51:32.769 DEBUG:teuthology.orchestra.run.smithi084:> set -e 2024-04-28T12:51:32.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-28T12:51:32.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-28T12:51:32.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-28T12:51:32.769 DEBUG:teuthology.orchestra.run.smithi084:> ') 2024-04-28T12:51:32.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:51:33.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:51:33.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:51:33.026 DEBUG:teuthology.orchestra.run.smithi084:> ip netns list 2024-04-28T12:51:33.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:51:33.078 DEBUG:teuthology.orchestra.run.smithi084:> ip netns list-id 2024-04-28T12:51:33.125 DEBUG:teuthology.orchestra.run.smithi084:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:51:33.125 DEBUG:teuthology.orchestra.run.smithi084:> set -e 2024-04-28T12:51:33.125 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-28T12:51:33.125 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-28T12:51:33.125 DEBUG:teuthology.orchestra.run.smithi084:> ') 2024-04-28T12:51:33.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:51:33.299 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:51:33.309 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-28T12:51:33.309 DEBUG:teuthology.orchestra.run.smithi084:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:51:33.309 DEBUG:teuthology.orchestra.run.smithi084:> set -e 2024-04-28T12:51:33.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-28T12:51:33.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-28T12:51:33.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-28T12:51:33.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-28T12:51:33.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-28T12:51:33.310 DEBUG:teuthology.orchestra.run.smithi084:> ') 2024-04-28T12:51:33.410 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:51:33.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:51:33.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:51:33.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:33.573 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:51:33.583 DEBUG:teuthology.orchestra.run.smithi084:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-28T12:51:33.583 DEBUG:teuthology.orchestra.run.smithi084:> set -e 2024-04-28T12:51:33.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip link set brx.0 up 2024-04-28T12:51:33.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip link set dev brx.0 master ceph-brx 2024-04-28T12:51:33.583 DEBUG:teuthology.orchestra.run.smithi084:> ') 2024-04-28T12:51:33.679 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:51:33.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:33 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:51:33.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:33 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:51:33.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:33 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:33.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:51:33.786 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-28T12:51:33.786 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-28T12:51:33.832 INFO:teuthology.orchestra.run.smithi084.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-28T12:51:33.832 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-28T12:51:33.833 DEBUG:teuthology.orchestra.run.smithi084:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-28T12:51:33.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:33 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:51:33.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:33 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:51:33.916 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:33 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:51:33.926 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-28T12:51:33.927 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-28T12:51:33.927 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/9aabb1be-843e-42a1-9aa4-4f80b25dd056 2024-04-28T12:51:33.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:51:33.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/9aabb1be-843e-42a1-9aa4-4f80b25dd056 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-04-28T12:51:34.053 INFO:teuthology.orchestra.run.smithi084.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-04-28T12:51:34.054 INFO:teuthology.orchestra.run.smithi084.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-04-28T12:51:34.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:34 smithi084 ceph-mon[28081]: pgmap v416: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:34.743 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:34 smithi161 ceph-mon[62387]: pgmap v416: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:34.814 INFO:teuthology.orchestra.run.smithi084.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-04-28T12:51:34.815 INFO:teuthology.orchestra.run.smithi084.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-04-28T12:51:34.815 INFO:teuthology.orchestra.run.smithi084.stdout:mount.ceph: resolved to: "172.21.15.84:3300,172.21.15.161:3300,172.21.15.179:3300" 2024-04-28T12:51:34.815 INFO:teuthology.orchestra.run.smithi084.stdout:mount.ceph: trying mount with new device syntax: 0@ddee604a-055b-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/9aabb1be-843e-42a1-9aa4-4f80b25dd056 2024-04-28T12:51:34.815 INFO:teuthology.orchestra.run.smithi084.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.84:3300/172.21.15.161:3300/172.21.15.179:3300" will pass to kernel 2024-04-28T12:51:34.821 INFO:teuthology.orchestra.run.smithi084.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-28T12:51:34.821 INFO:teuthology.orchestra.run.smithi084.stdout: You just mounted a file system that supports labels which does not 2024-04-28T12:51:34.821 INFO:teuthology.orchestra.run.smithi084.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-28T12:51:34.821 INFO:teuthology.orchestra.run.smithi084.stdout: applications will generate AVC messages and not be allowed access to 2024-04-28T12:51:34.821 INFO:teuthology.orchestra.run.smithi084.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-28T12:51:34.825 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-28T12:51:34.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T12:51:34.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-28T12:51:34.913 DEBUG:teuthology.orchestra.run.smithi084:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-28T12:51:34.915 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:34 smithi179 ceph-mon[63029]: pgmap v416: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:35.021 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-28T12:51:35.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:51:35.035 INFO:teuthology.orchestra.run.smithi084.stdout:client25394 2024-04-28T12:51:35.036 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T12:51:35.036 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T12:51:35.086 DEBUG:teuthology.orchestra.run.smithi084:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-28T12:51:35.189 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-28T12:51:35.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:51:35.203 INFO:teuthology.orchestra.run.smithi084.stdout:client25394 2024-04-28T12:51:35.203 DEBUG:teuthology.orchestra.run.smithi084:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-28T12:51:35.308 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-28T12:51:35.312 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:51:35.322 INFO:teuthology.orchestra.run.smithi084.stdout:client25394 2024-04-28T12:51:35.322 DEBUG:teuthology.orchestra.run.smithi084:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/ddee604a-055b-11ef-bc93-c7b262605968.client25394/status') 2024-04-28T12:51:35.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-28T12:51:35.452 INFO:teuthology.orchestra.run.smithi084.stdout:instance: client.25394 (3)172.21.15.84:0/2299411153 2024-04-28T12:51:35.453 INFO:teuthology.orchestra.run.smithi084.stdout:blocklisted: false 2024-04-28T12:51:35.453 INFO:teuthology.orchestra.run.smithi084.stderr:0+1 records in 2024-04-28T12:51:35.453 INFO:teuthology.orchestra.run.smithi084.stderr:0+1 records out 2024-04-28T12:51:35.453 INFO:teuthology.orchestra.run.smithi084.stderr:71 bytes copied, 7.5776e-05 s, 937 kB/s 2024-04-28T12:51:35.456 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-28T12:51:35.467 INFO:teuthology.run_tasks:Running task exec... 2024-04-28T12:51:35.483 INFO:teuthology.task.exec:Executing custom commands... 2024-04-28T12:51:35.484 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi084.front.sepia.ceph.com 2024-04-28T12:51:35.484 DEBUG:teuthology.orchestra.run.smithi084:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-28T12:51:35.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.684+0000 7fc04fe00640 1 -- 172.21.15.84:0/424170550 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0501065a0 msgr2=0x7fc05010ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:35.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.684+0000 7fc04fe00640 1 --2- 172.21.15.84:0/424170550 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0501065a0 0x7fc05010ce30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc048009ca0 tx=0x7fc04802a8c0 comp rx=0 tx=0).stop 2024-04-28T12:51:35.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.684+0000 7fc04fe00640 1 -- 172.21.15.84:0/424170550 shutdown_connections 2024-04-28T12:51:35.681 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.684+0000 7fc04fe00640 1 --2- 172.21.15.84:0/424170550 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0501065a0 0x7fc05010ce30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:35.682 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.684+0000 7fc04fe00640 1 --2- 172.21.15.84:0/424170550 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc050105be0 0x7fc050106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:35.682 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.684+0000 7fc04fe00640 1 --2- 172.21.15.84:0/424170550 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0501049f0 0x7fc050104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:35.682 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.684+0000 7fc04fe00640 1 -- 172.21.15.84:0/424170550 >> 172.21.15.84:0/424170550 conn(0x7fc0500fffa0 msgr2=0x7fc0501023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:35.682 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.685+0000 7fc04fe00640 1 -- 172.21.15.84:0/424170550 shutdown_connections 2024-04-28T12:51:35.682 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.685+0000 7fc04fe00640 1 -- 172.21.15.84:0/424170550 wait complete. 2024-04-28T12:51:35.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04fe00640 1 Processor -- start 2024-04-28T12:51:35.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04fe00640 1 -- start start 2024-04-28T12:51:35.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04fe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0501049f0 0x7fc05019abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:35.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc050105be0 0x7fc05019b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:35.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04fe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0501065a0 0x7fc0501a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:35.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04fe00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fc050110570 con 0x7fc0501049f0 2024-04-28T12:51:35.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04fe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc0501103f0 con 0x7fc050105be0 2024-04-28T12:51:35.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04fe00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fc0501106f0 con 0x7fc0501065a0 2024-04-28T12:51:35.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04e000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc050105be0 0x7fc05019b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:35.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04ea00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0501049f0 0x7fc05019abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:35.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04e000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc050105be0 0x7fc05019b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:39900/0 (socket says 172.21.15.84:39900) 2024-04-28T12:51:35.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04e000640 1 -- 172.21.15.84:0/2883617707 learned_addr learned my addr 172.21.15.84:0/2883617707 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:35.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.686+0000 7fc04f400640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0501065a0 0x7fc0501a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:35.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.687+0000 7fc04ea00640 1 -- 172.21.15.84:0/2883617707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0501065a0 msgr2=0x7fc0501a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:35.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.687+0000 7fc04ea00640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0501065a0 0x7fc0501a2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:35.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.687+0000 7fc04ea00640 1 -- 172.21.15.84:0/2883617707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc050105be0 msgr2=0x7fc05019b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:35.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.687+0000 7fc04ea00640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc050105be0 0x7fc05019b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:35.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.687+0000 7fc04ea00640 1 -- 172.21.15.84:0/2883617707 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0501a27d0 con 0x7fc0501049f0 2024-04-28T12:51:35.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.687+0000 7fc04ea00640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0501049f0 0x7fc05019abb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc04000e9c0 tx=0x7fc04000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:35.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.687+0000 7fc047400640 1 -- 172.21.15.84:0/2883617707 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04000ccc0 con 0x7fc0501049f0 2024-04-28T12:51:35.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.687+0000 7fc047400640 1 -- 172.21.15.84:0/2883617707 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0400035f0 con 0x7fc0501049f0 2024-04-28T12:51:35.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.687+0000 7fc047400640 1 -- 172.21.15.84:0/2883617707 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04000ce60 con 0x7fc0501049f0 2024-04-28T12:51:35.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.688+0000 7fc04fe00640 1 -- 172.21.15.84:0/2883617707 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0501a2ac0 con 0x7fc0501049f0 2024-04-28T12:51:35.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.688+0000 7fc04fe00640 1 -- 172.21.15.84:0/2883617707 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0501a2f90 con 0x7fc0501049f0 2024-04-28T12:51:35.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.689+0000 7fc04fe00640 1 -- 172.21.15.84:0/2883617707 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc05010bd10 con 0x7fc0501049f0 2024-04-28T12:51:35.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.691+0000 7fc047400640 1 -- 172.21.15.84:0/2883617707 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc040004000 con 0x7fc0501049f0 2024-04-28T12:51:35.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.692+0000 7fc047400640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc0340a3c30 0x7fc0340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:35.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.693+0000 7fc047400640 1 -- 172.21.15.84:0/2883617707 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fc0400d0e10 con 0x7fc0501049f0 2024-04-28T12:51:35.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.693+0000 7fc04e000640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc0340a3c30 0x7fc0340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:35.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.693+0000 7fc04e000640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc0340a3c30 0x7fc0340a60f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc05019c0c0 tx=0x7fc03c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:35.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.697+0000 7fc047400640 1 -- 172.21.15.84:0/2883617707 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc0400d4030 con 0x7fc0501049f0 2024-04-28T12:51:35.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:35.944+0000 7fc04fe00640 1 -- 172.21.15.84:0/2883617707 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fc05002dad0 con 0x7fc0501049f0 2024-04-28T12:51:36.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:35 smithi179 ceph-mon[63029]: pgmap v417: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:36.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:35 smithi084 ceph-mon[28081]: pgmap v417: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:36.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:35 smithi161 ceph-mon[62387]: pgmap v417: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:36.762 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.765+0000 7fc047400640 1 -- 172.21.15.84:0/2883617707 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7fc04009d2b0 con 0x7fc0501049f0 2024-04-28T12:51:36.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.769+0000 7fc04fe00640 1 -- 172.21.15.84:0/2883617707 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc0340a3c30 msgr2=0x7fc0340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:36.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.769+0000 7fc04fe00640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc0340a3c30 0x7fc0340a60f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc05019c0c0 tx=0x7fc03c005f90 comp rx=0 tx=0).stop 2024-04-28T12:51:36.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.769+0000 7fc04fe00640 1 -- 172.21.15.84:0/2883617707 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0501049f0 msgr2=0x7fc05019abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:36.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.769+0000 7fc04fe00640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0501049f0 0x7fc05019abb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc04000e9c0 tx=0x7fc04000ee80 comp rx=0 tx=0).stop 2024-04-28T12:51:36.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.769+0000 7fc04fe00640 1 -- 172.21.15.84:0/2883617707 shutdown_connections 2024-04-28T12:51:36.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.769+0000 7fc04fe00640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fc0340a3c30 0x7fc0340a60f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:36.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.769+0000 7fc04fe00640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0501065a0 0x7fc0501a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:36.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.769+0000 7fc04fe00640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc050105be0 0x7fc05019b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:36.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.769+0000 7fc04fe00640 1 --2- 172.21.15.84:0/2883617707 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0501049f0 0x7fc05019abb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:36.766 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.769+0000 7fc04fe00640 1 -- 172.21.15.84:0/2883617707 >> 172.21.15.84:0/2883617707 conn(0x7fc0500fffa0 msgr2=0x7fc050108ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:36.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.769+0000 7fc04fe00640 1 -- 172.21.15.84:0/2883617707 shutdown_connections 2024-04-28T12:51:36.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.769+0000 7fc04fe00640 1 -- 172.21.15.84:0/2883617707 wait complete. 2024-04-28T12:51:36.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-28T12:51:36.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.985+0000 7f08dd800640 1 -- 172.21.15.84:0/2790299161 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08d8075390 msgr2=0x7f08d8075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:36.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.985+0000 7f08dd800640 1 --2- 172.21.15.84:0/2790299161 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08d8075390 0x7f08d8075810 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009930 tx=0x7f08c002a420 comp rx=0 tx=0).stop 2024-04-28T12:51:36.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.985+0000 7f08dd800640 1 -- 172.21.15.84:0/2790299161 shutdown_connections 2024-04-28T12:51:36.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.985+0000 7f08dd800640 1 --2- 172.21.15.84:0/2790299161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08d810ed90 0x7f08d8111180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:36.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.985+0000 7f08dd800640 1 --2- 172.21.15.84:0/2790299161 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08d8075390 0x7f08d8075810 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:36.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.985+0000 7f08dd800640 1 --2- 172.21.15.84:0/2790299161 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08d8076970 0x7f08d8074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:36.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.985+0000 7f08dd800640 1 -- 172.21.15.84:0/2790299161 >> 172.21.15.84:0/2790299161 conn(0x7f08d80fdd30 msgr2=0x7f08d8100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T12:51:36.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.985+0000 7f08dd800640 1 -- 172.21.15.84:0/2790299161 shutdown_connections 2024-04-28T12:51:36.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.985+0000 7f08dd800640 1 -- 172.21.15.84:0/2790299161 wait complete. 2024-04-28T12:51:36.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.986+0000 7f08dd800640 1 Processor -- start 2024-04-28T12:51:36.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.987+0000 7f08dd800640 1 -- start start 2024-04-28T12:51:36.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.987+0000 7f08dd800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08d8075390 0x7f08d819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:36.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.987+0000 7f08dd800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08d8076970 0x7f08d819f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:36.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.987+0000 7f08dd800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08d810ed90 0x7f08d81a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:36.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.987+0000 7f08dd800640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f08d81148e0 con 0x7f08d810ed90 2024-04-28T12:51:36.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.987+0000 7f08dd800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f08d8114760 con 0x7f08d8076970 2024-04-28T12:51:36.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.987+0000 7f08dd800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f08d8114a60 con 0x7f08d8075390 2024-04-28T12:51:36.985 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.987+0000 7f08dce00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08d810ed90 0x7f08d81a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.987+0000 7f08d7e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08d8075390 0x7f08d819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.987+0000 7f08dce00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08d810ed90 0x7f08d81a65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52068/0 (socket says 172.21.15.84:52068) 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.987+0000 7f08dce00640 1 -- 172.21.15.84:0/2794133779 learned_addr learned my addr 172.21.15.84:0/2794133779 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.987+0000 7f08cfe00640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08d8076970 0x7f08d819f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.988+0000 7f08dce00640 1 -- 172.21.15.84:0/2794133779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08d8075390 msgr2=0x7f08d819f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.988+0000 7f08dce00640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08d8075390 0x7f08d819f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.988+0000 7f08dce00640 1 -- 172.21.15.84:0/2794133779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08d8076970 msgr2=0x7f08d819f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.988+0000 7f08dce00640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08d8076970 0x7f08d819f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.988+0000 7f08dce00640 1 -- 172.21.15.84:0/2794133779 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d81a6c50 con 0x7f08d810ed90 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.988+0000 7f08d7e00640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f08d8075390 0x7f08d819f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.988+0000 7f08dce00640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08d810ed90 0x7f08d81a65e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f08d000eb10 tx=0x7f08d000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.988+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d0004b20 con 0x7f08d810ed90 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.988+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08d0004cc0 con 0x7f08d810ed90 2024-04-28T12:51:36.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.988+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d0010670 con 0x7f08d810ed90 2024-04-28T12:51:36.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.988+0000 7f08dd800640 1 -- 172.21.15.84:0/2794133779 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d81a6f40 con 0x7f08d810ed90 2024-04-28T12:51:36.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.989+0000 7f08dd800640 1 -- 172.21.15.84:0/2794133779 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d81a7400 con 0x7f08d810ed90 2024-04-28T12:51:36.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.990+0000 7f08dd800640 1 -- 172.21.15.84:0/2794133779 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08d8104e50 con 0x7f08d810ed90 2024-04-28T12:51:36.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.991+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f08d0005020 con 0x7f08d810ed90 2024-04-28T12:51:36.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.993+0000 7f08d5600640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f08bc0a3cb0 0x7f08bc0a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T12:51:36.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.993+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f08d00d0e50 con 0x7f08d810ed90 2024-04-28T12:51:36.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.993+0000 7f08d7e00640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f08bc0a3cb0 0x7f08bc0a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T12:51:36.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.994+0000 7f08d7e00640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f08bc0a3cb0 0x7f08bc0a6170 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009fc0 tx=0x7f08c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T12:51:36.993 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:36.997+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f08d009d2f0 con 0x7f08d810ed90 2024-04-28T12:51:37.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:36 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2883617707' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-28T12:51:37.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:36 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2883617707' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-28T12:51:37.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:51:37.228+0000 7f08dd800640 1 -- 172.21.15.84:0/2794133779 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f08d8062e10 con 0x7f08bc0a3cb0 2024-04-28T12:51:37.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:36 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2883617707' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-28T12:51:38.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:37 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2883617707' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-28T12:51:38.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:37 smithi179 ceph-mon[63029]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T12:51:38.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:37 smithi179 ceph-mon[63029]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:38.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:37 smithi179 ceph-mon[63029]: pgmap v418: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:38.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:37 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2883617707' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-28T12:51:38.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:37 smithi084 ceph-mon[28081]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T12:51:38.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:37 smithi084 ceph-mon[28081]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:38.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:37 smithi084 ceph-mon[28081]: pgmap v418: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:38.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:37 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2883617707' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-28T12:51:38.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:37 smithi161 ceph-mon[62387]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-28T12:51:38.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:37 smithi161 ceph-mon[62387]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T12:51:38.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:37 smithi161 ceph-mon[62387]: pgmap v418: 129 pgs: 129 active+clean; 635 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:40.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:39 smithi179 ceph-mon[63029]: pgmap v419: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:51:40.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:39 smithi084 ceph-mon[28081]: pgmap v419: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:51:40.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:39 smithi161 ceph-mon[62387]: pgmap v419: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:51:41.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:41 smithi084 ceph-mon[28081]: pgmap v420: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-28T12:51:42.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:41 smithi179 ceph-mon[63029]: pgmap v420: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-28T12:51:42.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:41 smithi161 ceph-mon[62387]: pgmap v420: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-28T12:51:44.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:43 smithi179 ceph-mon[63029]: pgmap v421: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-28T12:51:44.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:43 smithi084 ceph-mon[28081]: pgmap v421: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-28T12:51:44.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:43 smithi161 ceph-mon[62387]: pgmap v421: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-28T12:51:46.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:45 smithi179 ceph-mon[63029]: pgmap v422: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:51:46.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:45 smithi084 ceph-mon[28081]: pgmap v422: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:51:46.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:45 smithi161 ceph-mon[62387]: pgmap v422: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:51:48.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:47 smithi179 ceph-mon[63029]: pgmap v423: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-28T12:51:48.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:47 smithi084 ceph-mon[28081]: pgmap v423: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-28T12:51:48.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:47 smithi161 ceph-mon[62387]: pgmap v423: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-28T12:51:50.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:49 smithi179 ceph-mon[63029]: pgmap v424: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:51:50.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:49 smithi084 ceph-mon[28081]: pgmap v424: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:51:50.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:49 smithi161 ceph-mon[62387]: pgmap v424: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-28T12:51:52.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:51 smithi179 ceph-mon[63029]: pgmap v425: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:52.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:51 smithi084 ceph-mon[28081]: pgmap v425: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:52.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:51 smithi161 ceph-mon[62387]: pgmap v425: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:51:54.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:53 smithi179 ceph-mon[63029]: pgmap v426: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:54.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:53 smithi084 ceph-mon[28081]: pgmap v426: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:54.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:53 smithi161 ceph-mon[62387]: pgmap v426: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:56.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:55 smithi179 ceph-mon[63029]: pgmap v427: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:56.167 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:55 smithi084 ceph-mon[28081]: pgmap v427: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:56.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:55 smithi161 ceph-mon[62387]: pgmap v427: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:51:58.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:57 smithi179 ceph-mon[63029]: pgmap v428: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:58.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:57 smithi084 ceph-mon[28081]: pgmap v428: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:51:58.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:57 smithi161 ceph-mon[62387]: pgmap v428: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:00.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:51:59 smithi179 ceph-mon[63029]: pgmap v429: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:00.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:51:59 smithi084 ceph-mon[28081]: pgmap v429: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:00.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:51:59 smithi161 ceph-mon[62387]: pgmap v429: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:02.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:01 smithi179 ceph-mon[63029]: pgmap v430: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:02.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:01 smithi084 ceph-mon[28081]: pgmap v430: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:02.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:01 smithi161 ceph-mon[62387]: pgmap v430: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:04.046 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:03 smithi161 ceph-mon[62387]: pgmap v431: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:04.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:03 smithi179 ceph-mon[63029]: pgmap v431: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:04.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:03 smithi084 ceph-mon[28081]: pgmap v431: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:06.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:05 smithi179 ceph-mon[63029]: pgmap v432: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:06.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:05 smithi084 ceph-mon[28081]: pgmap v432: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:06.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:05 smithi161 ceph-mon[62387]: pgmap v432: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:08.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:07 smithi179 ceph-mon[63029]: pgmap v433: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:08.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:07 smithi084 ceph-mon[28081]: pgmap v433: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:08.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:07 smithi161 ceph-mon[62387]: pgmap v433: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:10.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:09 smithi179 ceph-mon[63029]: pgmap v434: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:10.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:09 smithi084 ceph-mon[28081]: pgmap v434: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:10.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:09 smithi161 ceph-mon[62387]: pgmap v434: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:12.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:11 smithi179 ceph-mon[63029]: pgmap v435: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:12.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:11 smithi084 ceph-mon[28081]: pgmap v435: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:12.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:11 smithi161 ceph-mon[62387]: pgmap v435: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:14.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:13 smithi179 ceph-mon[63029]: pgmap v436: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:14.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:13 smithi084 ceph-mon[28081]: pgmap v436: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:14.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:13 smithi161 ceph-mon[62387]: pgmap v436: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:16.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:15 smithi179 ceph-mon[63029]: pgmap v437: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:16.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:15 smithi084 ceph-mon[28081]: pgmap v437: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:16.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:15 smithi161 ceph-mon[62387]: pgmap v437: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:18.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:17 smithi179 ceph-mon[63029]: pgmap v438: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:18.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:17 smithi084 ceph-mon[28081]: pgmap v438: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:18.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:17 smithi161 ceph-mon[62387]: pgmap v438: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:20.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:19 smithi179 ceph-mon[63029]: pgmap v439: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:20.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:19 smithi084 ceph-mon[28081]: pgmap v439: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:20.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:19 smithi161 ceph-mon[62387]: pgmap v439: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:22.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:21 smithi179 ceph-mon[63029]: pgmap v440: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:22.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:21 smithi084 ceph-mon[28081]: pgmap v440: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:22.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:21 smithi161 ceph-mon[62387]: pgmap v440: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:24.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:23 smithi179 ceph-mon[63029]: pgmap v441: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:24.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:23 smithi084 ceph-mon[28081]: pgmap v441: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:24.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:23 smithi161 ceph-mon[62387]: pgmap v441: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:27.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:26 smithi179 ceph-mon[63029]: pgmap v442: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:27.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:26 smithi084 ceph-mon[28081]: pgmap v442: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:27.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:26 smithi161 ceph-mon[62387]: pgmap v442: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:29.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:28 smithi179 ceph-mon[63029]: pgmap v443: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:29.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:28 smithi084 ceph-mon[28081]: pgmap v443: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:29.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:28 smithi161 ceph-mon[62387]: pgmap v443: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:31.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:30 smithi179 ceph-mon[63029]: pgmap v444: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:31.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:30 smithi084 ceph-mon[28081]: pgmap v444: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:31.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:30 smithi161 ceph-mon[62387]: pgmap v444: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:33.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:32 smithi179 ceph-mon[63029]: pgmap v445: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:33.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:32 smithi084 ceph-mon[28081]: pgmap v445: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:33.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:32 smithi161 ceph-mon[62387]: pgmap v445: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:34.145 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:33 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:52:34.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:33 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:52:34.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:33 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:52:35.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:34 smithi179 ceph-mon[63029]: pgmap v446: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:35.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:52:35.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:52:35.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:52:35.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:34 smithi084 ceph-mon[28081]: pgmap v446: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:35.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:52:35.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:52:35.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:52:35.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:34 smithi161 ceph-mon[62387]: pgmap v446: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:35.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:52:35.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:52:35.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:52:37.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:36 smithi179 ceph-mon[63029]: pgmap v447: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:37.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:36 smithi084 ceph-mon[28081]: pgmap v447: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:37.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:36 smithi161 ceph-mon[62387]: pgmap v447: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:39.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:38 smithi179 ceph-mon[63029]: pgmap v448: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:39.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:38 smithi084 ceph-mon[28081]: pgmap v448: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:39.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:38 smithi161 ceph-mon[62387]: pgmap v448: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:41.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:40 smithi179 ceph-mon[63029]: pgmap v449: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:41.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:40 smithi084 ceph-mon[28081]: pgmap v449: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:41.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:40 smithi161 ceph-mon[62387]: pgmap v449: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:43.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:42 smithi179 ceph-mon[63029]: pgmap v450: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:43.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:42 smithi084 ceph-mon[28081]: pgmap v450: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:43.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:42 smithi161 ceph-mon[62387]: pgmap v450: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:44.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:44 smithi084 ceph-mon[28081]: pgmap v451: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:45.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:44 smithi179 ceph-mon[63029]: pgmap v451: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:45.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:44 smithi161 ceph-mon[62387]: pgmap v451: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:47.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:46 smithi179 ceph-mon[63029]: pgmap v452: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:47.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:46 smithi084 ceph-mon[28081]: pgmap v452: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:47.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:46 smithi161 ceph-mon[62387]: pgmap v452: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:49.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:48 smithi179 ceph-mon[63029]: pgmap v453: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:49.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:48 smithi084 ceph-mon[28081]: pgmap v453: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:49.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:48 smithi161 ceph-mon[62387]: pgmap v453: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:51.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:50 smithi179 ceph-mon[63029]: pgmap v454: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:51.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:50 smithi084 ceph-mon[28081]: pgmap v454: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:51.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:50 smithi161 ceph-mon[62387]: pgmap v454: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:52:53.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:52 smithi179 ceph-mon[63029]: pgmap v455: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:53.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:52 smithi084 ceph-mon[28081]: pgmap v455: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:53.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:52 smithi161 ceph-mon[62387]: pgmap v455: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:52:55.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:54 smithi179 ceph-mon[63029]: pgmap v456: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:55.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:54 smithi084 ceph-mon[28081]: pgmap v456: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:55.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:54 smithi161 ceph-mon[62387]: pgmap v456: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:57.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:56 smithi179 ceph-mon[63029]: pgmap v457: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:57.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:56 smithi084 ceph-mon[28081]: pgmap v457: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:57.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:56 smithi161 ceph-mon[62387]: pgmap v457: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:52:59.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:52:58 smithi179 ceph-mon[63029]: pgmap v458: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:59.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:52:58 smithi084 ceph-mon[28081]: pgmap v458: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:52:59.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:52:58 smithi161 ceph-mon[62387]: pgmap v458: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:01.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:00 smithi179 ceph-mon[63029]: pgmap v459: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:01.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:00 smithi084 ceph-mon[28081]: pgmap v459: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:01.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:00 smithi161 ceph-mon[62387]: pgmap v459: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:03.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:02 smithi179 ceph-mon[63029]: pgmap v460: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:03.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:02 smithi084 ceph-mon[28081]: pgmap v460: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:03.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:02 smithi161 ceph-mon[62387]: pgmap v460: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:05.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:04 smithi179 ceph-mon[63029]: pgmap v461: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:05.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:04 smithi084 ceph-mon[28081]: pgmap v461: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:05.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:04 smithi161 ceph-mon[62387]: pgmap v461: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:07.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:06 smithi179 ceph-mon[63029]: pgmap v462: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:07.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:06 smithi084 ceph-mon[28081]: pgmap v462: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:07.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:06 smithi161 ceph-mon[62387]: pgmap v462: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:09.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:08 smithi179 ceph-mon[63029]: pgmap v463: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:09.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:08 smithi084 ceph-mon[28081]: pgmap v463: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:09.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:08 smithi161 ceph-mon[62387]: pgmap v463: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:11.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:10 smithi179 ceph-mon[63029]: pgmap v464: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:11.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:10 smithi084 ceph-mon[28081]: pgmap v464: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:11.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:10 smithi161 ceph-mon[62387]: pgmap v464: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:13.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:12 smithi179 ceph-mon[63029]: pgmap v465: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:13.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:12 smithi084 ceph-mon[28081]: pgmap v465: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:13.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:12 smithi161 ceph-mon[62387]: pgmap v465: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:15.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:14 smithi179 ceph-mon[63029]: pgmap v466: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:15.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:14 smithi084 ceph-mon[28081]: pgmap v466: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:15.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:14 smithi161 ceph-mon[62387]: pgmap v466: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:17.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:16 smithi179 ceph-mon[63029]: pgmap v467: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:17.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:16 smithi084 ceph-mon[28081]: pgmap v467: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:17.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:16 smithi161 ceph-mon[62387]: pgmap v467: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:19.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:18 smithi179 ceph-mon[63029]: pgmap v468: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:19.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:18 smithi084 ceph-mon[28081]: pgmap v468: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:19.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:18 smithi161 ceph-mon[62387]: pgmap v468: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:21.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:20 smithi179 ceph-mon[63029]: pgmap v469: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:21.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:20 smithi084 ceph-mon[28081]: pgmap v469: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:21.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:20 smithi161 ceph-mon[62387]: pgmap v469: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:23.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:22 smithi179 ceph-mon[63029]: pgmap v470: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:23.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:22 smithi084 ceph-mon[28081]: pgmap v470: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:23.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:22 smithi161 ceph-mon[62387]: pgmap v470: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:25.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:24 smithi179 ceph-mon[63029]: pgmap v471: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:25.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:24 smithi084 ceph-mon[28081]: pgmap v471: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:25.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:24 smithi161 ceph-mon[62387]: pgmap v471: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:27.166 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:26 smithi179 ceph-mon[63029]: pgmap v472: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:27.168 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:26 smithi084 ceph-mon[28081]: pgmap v472: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:27.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:26 smithi161 ceph-mon[62387]: pgmap v472: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:29.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:28 smithi161 ceph-mon[62387]: pgmap v473: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:29.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:28 smithi179 ceph-mon[63029]: pgmap v473: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:29.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:28 smithi084 ceph-mon[28081]: pgmap v473: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:31.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:30 smithi161 ceph-mon[62387]: pgmap v474: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:31.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:30 smithi179 ceph-mon[63029]: pgmap v474: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:31.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:30 smithi084 ceph-mon[28081]: pgmap v474: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:33.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:32 smithi161 ceph-mon[62387]: pgmap v475: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:33.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:32 smithi179 ceph-mon[63029]: pgmap v475: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:33.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:32 smithi084 ceph-mon[28081]: pgmap v475: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:35.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:34 smithi161 ceph-mon[62387]: pgmap v476: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:35.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:34 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:53:35.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:34 smithi179 ceph-mon[63029]: pgmap v476: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:35.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:34 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:53:35.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:34 smithi084 ceph-mon[28081]: pgmap v476: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:35.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:34 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:53:36.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:35 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:53:36.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:35 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:53:36.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:35 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:53:36.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:35 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:53:36.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:35 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:53:36.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:35 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:53:36.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:35 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:53:36.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:35 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:53:36.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:35 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:53:37.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:36 smithi161 ceph-mon[62387]: pgmap v477: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:37.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:36 smithi179 ceph-mon[63029]: pgmap v477: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:37.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:36 smithi084 ceph-mon[28081]: pgmap v477: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:39.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:38 smithi161 ceph-mon[62387]: pgmap v478: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:39.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:38 smithi179 ceph-mon[63029]: pgmap v478: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:39.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:38 smithi084 ceph-mon[28081]: pgmap v478: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:41.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:40 smithi161 ceph-mon[62387]: pgmap v479: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:41.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:40 smithi179 ceph-mon[63029]: pgmap v479: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:41.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:40 smithi084 ceph-mon[28081]: pgmap v479: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:43.221 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:42 smithi084 ceph-mon[28081]: pgmap v480: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:43.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:42 smithi161 ceph-mon[62387]: pgmap v480: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:43.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:42 smithi179 ceph-mon[63029]: pgmap v480: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:45.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:44 smithi161 ceph-mon[62387]: pgmap v481: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:45.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:44 smithi179 ceph-mon[63029]: pgmap v481: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:45.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:44 smithi084 ceph-mon[28081]: pgmap v481: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:47.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:46 smithi161 ceph-mon[62387]: pgmap v482: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:47.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:46 smithi179 ceph-mon[63029]: pgmap v482: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:47.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:46 smithi084 ceph-mon[28081]: pgmap v482: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:49.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:48 smithi161 ceph-mon[62387]: pgmap v483: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:49.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:48 smithi179 ceph-mon[63029]: pgmap v483: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:49.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:48 smithi084 ceph-mon[28081]: pgmap v483: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:51.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:50 smithi161 ceph-mon[62387]: pgmap v484: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:51.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:50 smithi179 ceph-mon[63029]: pgmap v484: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:51.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:50 smithi084 ceph-mon[28081]: pgmap v484: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:53:53.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:52 smithi161 ceph-mon[62387]: pgmap v485: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:53.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:52 smithi179 ceph-mon[63029]: pgmap v485: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:53.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:52 smithi084 ceph-mon[28081]: pgmap v485: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:53:55.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:54 smithi161 ceph-mon[62387]: pgmap v486: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:55.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:54 smithi179 ceph-mon[63029]: pgmap v486: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:55.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:54 smithi084 ceph-mon[28081]: pgmap v486: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:57.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:56 smithi179 ceph-mon[63029]: pgmap v487: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:57.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:56 smithi084 ceph-mon[28081]: pgmap v487: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:57.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:57 smithi161 ceph-mon[62387]: pgmap v487: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:53:59.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:53:59 smithi179 ceph-mon[63029]: pgmap v488: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:59.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:53:59 smithi084 ceph-mon[28081]: pgmap v488: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:53:59.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:53:59 smithi161 ceph-mon[62387]: pgmap v488: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:01.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:01 smithi179 ceph-mon[63029]: pgmap v489: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:01.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:01 smithi084 ceph-mon[28081]: pgmap v489: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:01.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:01 smithi161 ceph-mon[62387]: pgmap v489: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:03.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:03 smithi179 ceph-mon[63029]: pgmap v490: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:03.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:03 smithi084 ceph-mon[28081]: pgmap v490: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:03.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:03 smithi161 ceph-mon[62387]: pgmap v490: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:05.297 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:05 smithi161 ceph-mon[62387]: pgmap v491: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:05.416 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:05 smithi179 ceph-mon[63029]: pgmap v491: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:05.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:05 smithi084 ceph-mon[28081]: pgmap v491: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:07.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:07 smithi179 ceph-mon[63029]: pgmap v492: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:07.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:07 smithi084 ceph-mon[28081]: pgmap v492: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:07.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:07 smithi161 ceph-mon[62387]: pgmap v492: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:09.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:09 smithi179 ceph-mon[63029]: pgmap v493: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:09.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:09 smithi084 ceph-mon[28081]: pgmap v493: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:09.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:09 smithi161 ceph-mon[62387]: pgmap v493: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:11.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:11 smithi179 ceph-mon[63029]: pgmap v494: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:11.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:11 smithi084 ceph-mon[28081]: pgmap v494: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:11.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:11 smithi161 ceph-mon[62387]: pgmap v494: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:13.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:13 smithi179 ceph-mon[63029]: pgmap v495: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:13.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:13 smithi084 ceph-mon[28081]: pgmap v495: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:13.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:13 smithi161 ceph-mon[62387]: pgmap v495: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:15.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:15 smithi179 ceph-mon[63029]: pgmap v496: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:15.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:15 smithi084 ceph-mon[28081]: pgmap v496: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:15.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:15 smithi161 ceph-mon[62387]: pgmap v496: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:17.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:17 smithi179 ceph-mon[63029]: pgmap v497: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:17.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:17 smithi084 ceph-mon[28081]: pgmap v497: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:17.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:17 smithi161 ceph-mon[62387]: pgmap v497: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:19.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:19 smithi179 ceph-mon[63029]: pgmap v498: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:19.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:19 smithi084 ceph-mon[28081]: pgmap v498: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:19.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:19 smithi161 ceph-mon[62387]: pgmap v498: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:21.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:21 smithi179 ceph-mon[63029]: pgmap v499: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:21.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:21 smithi084 ceph-mon[28081]: pgmap v499: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:21.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:21 smithi161 ceph-mon[62387]: pgmap v499: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:23.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:23 smithi179 ceph-mon[63029]: pgmap v500: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:23.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:23 smithi084 ceph-mon[28081]: pgmap v500: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:23.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:23 smithi161 ceph-mon[62387]: pgmap v500: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:25.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:25 smithi179 ceph-mon[63029]: pgmap v501: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:25.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:25 smithi084 ceph-mon[28081]: pgmap v501: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:25.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:25 smithi161 ceph-mon[62387]: pgmap v501: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:27.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:27 smithi179 ceph-mon[63029]: pgmap v502: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:27.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:27 smithi084 ceph-mon[28081]: pgmap v502: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:27.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:27 smithi161 ceph-mon[62387]: pgmap v502: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:29.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:29 smithi179 ceph-mon[63029]: pgmap v503: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:29.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:29 smithi084 ceph-mon[28081]: pgmap v503: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:29.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:29 smithi161 ceph-mon[62387]: pgmap v503: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:31.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:31 smithi179 ceph-mon[63029]: pgmap v504: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:31.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:31 smithi084 ceph-mon[28081]: pgmap v504: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:31.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:31 smithi161 ceph-mon[62387]: pgmap v504: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:33.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:33 smithi179 ceph-mon[63029]: pgmap v505: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:33.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:33 smithi084 ceph-mon[28081]: pgmap v505: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:33.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:33 smithi161 ceph-mon[62387]: pgmap v505: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:35.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:35 smithi179 ceph-mon[63029]: pgmap v506: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:35.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:35 smithi084 ceph-mon[28081]: pgmap v506: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:35.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:35 smithi161 ceph-mon[62387]: pgmap v506: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:37.013 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:54:37.014+0000 7f08cf400640 1 -- 172.21.15.84:0/2794133779 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f08b00034c0 con 0x7f08d810ed90 2024-04-28T12:54:37.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:54:37.015+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 <== mon.0 v2:172.21.15.84:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f08b00034c0 con 0x7f08d810ed90 2024-04-28T12:54:37.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:37 smithi179 ceph-mon[63029]: pgmap v507: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:37.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:37 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:54:37.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:37 smithi084 ceph-mon[28081]: pgmap v507: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:37.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:37 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:54:37.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:37 smithi161 ceph-mon[62387]: pgmap v507: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:37.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:37 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:54:38.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:54:38.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:54:38.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:54:38.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:54:38.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:54:38.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:54:38.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:54:38.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:54:38.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:54:39.744 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:39 smithi161 ceph-mon[62387]: pgmap v508: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:39.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:39 smithi179 ceph-mon[63029]: pgmap v508: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:39.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:39 smithi084 ceph-mon[28081]: pgmap v508: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:41.744 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:41 smithi161 ceph-mon[62387]: pgmap v509: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:41 smithi179 ceph-mon[63029]: pgmap v509: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:41.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:41 smithi084 ceph-mon[28081]: pgmap v509: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:43.722 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:43 smithi084 ceph-mon[28081]: pgmap v510: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:43.744 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:43 smithi161 ceph-mon[62387]: pgmap v510: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:43.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:43 smithi179 ceph-mon[63029]: pgmap v510: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:45.744 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:45 smithi161 ceph-mon[62387]: pgmap v511: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:45.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:45 smithi179 ceph-mon[63029]: pgmap v511: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:45.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:45 smithi084 ceph-mon[28081]: pgmap v511: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:47.744 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:47 smithi161 ceph-mon[62387]: pgmap v512: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:47.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:47 smithi179 ceph-mon[63029]: pgmap v512: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:47.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:47 smithi084 ceph-mon[28081]: pgmap v512: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:49.719 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:49 smithi179 ceph-mon[63029]: pgmap v513: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:49.744 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:49 smithi161 ceph-mon[62387]: pgmap v513: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:49.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:49 smithi084 ceph-mon[28081]: pgmap v513: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:51.744 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:51 smithi161 ceph-mon[62387]: pgmap v514: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:51.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:51 smithi179 ceph-mon[63029]: pgmap v514: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:51.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:51 smithi084 ceph-mon[28081]: pgmap v514: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:54:53.744 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:53 smithi161 ceph-mon[62387]: pgmap v515: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:53.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:53 smithi179 ceph-mon[63029]: pgmap v515: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:53.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:53 smithi084 ceph-mon[28081]: pgmap v515: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:54:55.744 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:55 smithi161 ceph-mon[62387]: pgmap v516: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:55.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:55 smithi179 ceph-mon[63029]: pgmap v516: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:55.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:55 smithi084 ceph-mon[28081]: pgmap v516: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:57.744 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:57 smithi161 ceph-mon[62387]: pgmap v517: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:57.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:57 smithi179 ceph-mon[63029]: pgmap v517: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:57.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:57 smithi084 ceph-mon[28081]: pgmap v517: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:54:59.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:54:59 smithi179 ceph-mon[63029]: pgmap v518: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:59.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:54:59 smithi084 ceph-mon[28081]: pgmap v518: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:54:59.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:54:59 smithi161 ceph-mon[62387]: pgmap v518: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:01.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:01 smithi179 ceph-mon[63029]: pgmap v519: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:01.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:01 smithi084 ceph-mon[28081]: pgmap v519: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:01.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:01 smithi161 ceph-mon[62387]: pgmap v519: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:03.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:03 smithi179 ceph-mon[63029]: pgmap v520: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:03 smithi084 ceph-mon[28081]: pgmap v520: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:03.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:03 smithi161 ceph-mon[62387]: pgmap v520: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:05.797 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:05 smithi161 ceph-mon[62387]: pgmap v521: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:05.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:05 smithi179 ceph-mon[63029]: pgmap v521: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:05.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:05 smithi084 ceph-mon[28081]: pgmap v521: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:07.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:07 smithi179 ceph-mon[63029]: pgmap v522: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:07.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:07 smithi084 ceph-mon[28081]: pgmap v522: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:07.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:07 smithi161 ceph-mon[62387]: pgmap v522: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:09.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:09 smithi179 ceph-mon[63029]: pgmap v523: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:09.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:09 smithi084 ceph-mon[28081]: pgmap v523: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:09.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:09 smithi161 ceph-mon[62387]: pgmap v523: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:11.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:11 smithi179 ceph-mon[63029]: pgmap v524: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:11.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:11 smithi084 ceph-mon[28081]: pgmap v524: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:11.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:11 smithi161 ceph-mon[62387]: pgmap v524: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:13.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:13 smithi179 ceph-mon[63029]: pgmap v525: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:13.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:13 smithi084 ceph-mon[28081]: pgmap v525: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:13.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:13 smithi161 ceph-mon[62387]: pgmap v525: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:15.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:15 smithi179 ceph-mon[63029]: pgmap v526: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:15.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:15 smithi084 ceph-mon[28081]: pgmap v526: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:15.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:15 smithi161 ceph-mon[62387]: pgmap v526: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:17.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:17 smithi179 ceph-mon[63029]: pgmap v527: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:17.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:17 smithi084 ceph-mon[28081]: pgmap v527: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:17.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:17 smithi161 ceph-mon[62387]: pgmap v527: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:19.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:19 smithi179 ceph-mon[63029]: pgmap v528: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:19.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:19 smithi084 ceph-mon[28081]: pgmap v528: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:19.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:19 smithi161 ceph-mon[62387]: pgmap v528: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:21.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:21 smithi179 ceph-mon[63029]: pgmap v529: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:21.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:21 smithi084 ceph-mon[28081]: pgmap v529: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:21.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:21 smithi161 ceph-mon[62387]: pgmap v529: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:23.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:23 smithi179 ceph-mon[63029]: pgmap v530: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:23.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:23 smithi084 ceph-mon[28081]: pgmap v530: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:23.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:23 smithi161 ceph-mon[62387]: pgmap v530: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:25.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:25 smithi179 ceph-mon[63029]: pgmap v531: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:25.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:25 smithi084 ceph-mon[28081]: pgmap v531: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:25.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:25 smithi161 ceph-mon[62387]: pgmap v531: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:27.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:27 smithi179 ceph-mon[63029]: pgmap v532: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:27.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:27 smithi084 ceph-mon[28081]: pgmap v532: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:27.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:27 smithi161 ceph-mon[62387]: pgmap v532: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:29.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:29 smithi179 ceph-mon[63029]: pgmap v533: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:29.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:29 smithi084 ceph-mon[28081]: pgmap v533: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:29.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:29 smithi161 ceph-mon[62387]: pgmap v533: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:31.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:31 smithi179 ceph-mon[63029]: pgmap v534: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:31.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:31 smithi084 ceph-mon[28081]: pgmap v534: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:31.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:31 smithi161 ceph-mon[62387]: pgmap v534: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:33.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:33 smithi179 ceph-mon[63029]: pgmap v535: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:33.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:33 smithi084 ceph-mon[28081]: pgmap v535: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:33.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:33 smithi161 ceph-mon[62387]: pgmap v535: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:35.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:35 smithi179 ceph-mon[63029]: pgmap v536: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:35.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:35 smithi084 ceph-mon[28081]: pgmap v536: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:35.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:35 smithi161 ceph-mon[62387]: pgmap v536: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:37.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:37 smithi179 ceph-mon[63029]: pgmap v537: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:37.918 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:37 smithi084 ceph-mon[28081]: pgmap v537: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:37.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:37 smithi161 ceph-mon[62387]: pgmap v537: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:38.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:38 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:55:38.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:38 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:55:38.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:38 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:55:39.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:39 smithi179 ceph-mon[63029]: pgmap v538: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:39.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:55:39.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:55:39.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:55:39.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:39 smithi084 ceph-mon[28081]: pgmap v538: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:39.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:55:39.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:55:39.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:55:39.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:39 smithi161 ceph-mon[62387]: pgmap v538: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:39.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:39 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:55:39.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:39 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:55:39.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:39 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:55:41.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:41 smithi179 ceph-mon[63029]: pgmap v539: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:41.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:41 smithi084 ceph-mon[28081]: pgmap v539: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:41.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:41 smithi161 ceph-mon[62387]: pgmap v539: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:43.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:43 smithi179 ceph-mon[63029]: pgmap v540: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:43.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:43 smithi084 ceph-mon[28081]: pgmap v540: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:43.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:43 smithi161 ceph-mon[62387]: pgmap v540: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:45.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:45 smithi179 ceph-mon[63029]: pgmap v541: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:45.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:45 smithi084 ceph-mon[28081]: pgmap v541: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:45.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:45 smithi161 ceph-mon[62387]: pgmap v541: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:47.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:47 smithi179 ceph-mon[63029]: pgmap v542: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:47.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:47 smithi084 ceph-mon[28081]: pgmap v542: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:47.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:47 smithi161 ceph-mon[62387]: pgmap v542: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:49.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:49 smithi179 ceph-mon[63029]: pgmap v543: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:49.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:49 smithi084 ceph-mon[28081]: pgmap v543: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:49.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:49 smithi161 ceph-mon[62387]: pgmap v543: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:51.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:51 smithi179 ceph-mon[63029]: pgmap v544: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:51.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:51 smithi084 ceph-mon[28081]: pgmap v544: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:51.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:51 smithi161 ceph-mon[62387]: pgmap v544: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:55:53.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:53 smithi179 ceph-mon[63029]: pgmap v545: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:53.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:53 smithi084 ceph-mon[28081]: pgmap v545: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:53.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:53 smithi161 ceph-mon[62387]: pgmap v545: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:55:55.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:55 smithi179 ceph-mon[63029]: pgmap v546: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:55.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:55 smithi084 ceph-mon[28081]: pgmap v546: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:55.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:55 smithi161 ceph-mon[62387]: pgmap v546: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:55:57.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:57 smithi179 ceph-mon[63029]: pgmap v547: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:57.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:57 smithi084 ceph-mon[28081]: pgmap v547: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:57.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:57 smithi161 ceph-mon[62387]: pgmap v547: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:55:59.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:55:59 smithi161 ceph-mon[62387]: pgmap v548: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:00.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:55:59 smithi179 ceph-mon[63029]: pgmap v548: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:00.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:55:59 smithi084 ceph-mon[28081]: pgmap v548: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:01.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:01 smithi161 ceph-mon[62387]: pgmap v549: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:02.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:01 smithi179 ceph-mon[63029]: pgmap v549: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:02.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:01 smithi084 ceph-mon[28081]: pgmap v549: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:03.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:03 smithi161 ceph-mon[62387]: pgmap v550: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:04.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:03 smithi179 ceph-mon[63029]: pgmap v550: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:04.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:03 smithi084 ceph-mon[28081]: pgmap v550: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:05.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:05 smithi161 ceph-mon[62387]: pgmap v551: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:06.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:05 smithi179 ceph-mon[63029]: pgmap v551: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:06.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:05 smithi084 ceph-mon[28081]: pgmap v551: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:07.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:07 smithi161 ceph-mon[62387]: pgmap v552: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:08.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:07 smithi179 ceph-mon[63029]: pgmap v552: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:08.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:07 smithi084 ceph-mon[28081]: pgmap v552: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:09.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:09 smithi161 ceph-mon[62387]: pgmap v553: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:10.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:09 smithi179 ceph-mon[63029]: pgmap v553: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:10.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:09 smithi084 ceph-mon[28081]: pgmap v553: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:11.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:11 smithi161 ceph-mon[62387]: pgmap v554: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:12.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:11 smithi179 ceph-mon[63029]: pgmap v554: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:12.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:11 smithi084 ceph-mon[28081]: pgmap v554: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:13.972 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:13 smithi084 ceph-mon[28081]: pgmap v555: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:13.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:13 smithi161 ceph-mon[62387]: pgmap v555: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:14.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:13 smithi179 ceph-mon[63029]: pgmap v555: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:15.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:15 smithi161 ceph-mon[62387]: pgmap v556: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:16.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:15 smithi179 ceph-mon[63029]: pgmap v556: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:16.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:15 smithi084 ceph-mon[28081]: pgmap v556: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:17.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:17 smithi161 ceph-mon[62387]: pgmap v557: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:18.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:17 smithi179 ceph-mon[63029]: pgmap v557: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:18.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:17 smithi084 ceph-mon[28081]: pgmap v557: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:19.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:19 smithi161 ceph-mon[62387]: pgmap v558: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:20.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:19 smithi179 ceph-mon[63029]: pgmap v558: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:20.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:19 smithi084 ceph-mon[28081]: pgmap v558: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:21.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:21 smithi161 ceph-mon[62387]: pgmap v559: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:22.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:21 smithi179 ceph-mon[63029]: pgmap v559: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:22.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:21 smithi084 ceph-mon[28081]: pgmap v559: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:23.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:23 smithi161 ceph-mon[62387]: pgmap v560: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:24.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:23 smithi179 ceph-mon[63029]: pgmap v560: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:24.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:23 smithi084 ceph-mon[28081]: pgmap v560: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:25.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:25 smithi161 ceph-mon[62387]: pgmap v561: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:26.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:25 smithi179 ceph-mon[63029]: pgmap v561: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:26.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:25 smithi084 ceph-mon[28081]: pgmap v561: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:28.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:27 smithi179 ceph-mon[63029]: pgmap v562: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:28.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:27 smithi084 ceph-mon[28081]: pgmap v562: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:28.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:27 smithi161 ceph-mon[62387]: pgmap v562: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:30.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:29 smithi179 ceph-mon[63029]: pgmap v563: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:30.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:29 smithi084 ceph-mon[28081]: pgmap v563: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:30.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:29 smithi161 ceph-mon[62387]: pgmap v563: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:32.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:31 smithi179 ceph-mon[63029]: pgmap v564: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:32.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:31 smithi084 ceph-mon[28081]: pgmap v564: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:32.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:31 smithi161 ceph-mon[62387]: pgmap v564: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:33.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:32 smithi179 ceph-mon[63029]: pgmap v565: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:33.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:32 smithi084 ceph-mon[28081]: pgmap v565: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:33.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:32 smithi161 ceph-mon[62387]: pgmap v565: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:35.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:34 smithi179 ceph-mon[63029]: pgmap v566: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:35.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:34 smithi084 ceph-mon[28081]: pgmap v566: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:35.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:34 smithi161 ceph-mon[62387]: pgmap v566: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:37.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:36 smithi179 ceph-mon[63029]: pgmap v567: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:37.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:36 smithi084 ceph-mon[28081]: pgmap v567: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:37.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:36 smithi161 ceph-mon[62387]: pgmap v567: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:39.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:38 smithi179 ceph-mon[63029]: pgmap v568: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T12:56:39.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:38 smithi084 ceph-mon[28081]: pgmap v568: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T12:56:39.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:38 smithi161 ceph-mon[62387]: pgmap v568: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T12:56:40.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:39 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:56:40.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:39 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:56:40.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:39 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:56:41.167 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:40 smithi179 ceph-mon[63029]: pgmap v569: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:56:41.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:40 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:56:41.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:40 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:56:41.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:40 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:56:41.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:40 smithi084 ceph-mon[28081]: pgmap v569: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:56:41.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:40 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:56:41.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:40 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:56:41.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:40 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:56:41.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:40 smithi161 ceph-mon[62387]: pgmap v569: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:56:41.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:40 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:56:41.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:40 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:56:41.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:40 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:56:43.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:42 smithi084 ceph-mon[28081]: pgmap v570: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:56:43.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:42 smithi161 ceph-mon[62387]: pgmap v570: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:56:43.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:42 smithi179 ceph-mon[63029]: pgmap v570: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T12:56:45.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:44 smithi161 ceph-mon[62387]: pgmap v571: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T12:56:45.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:44 smithi179 ceph-mon[63029]: pgmap v571: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T12:56:45.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:44 smithi084 ceph-mon[28081]: pgmap v571: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T12:56:47.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:46 smithi161 ceph-mon[62387]: pgmap v572: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:47.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:46 smithi179 ceph-mon[63029]: pgmap v572: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:47.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:46 smithi084 ceph-mon[28081]: pgmap v572: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:49.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:48 smithi179 ceph-mon[63029]: pgmap v573: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:49.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:48 smithi161 ceph-mon[62387]: pgmap v573: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:49.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:48 smithi084 ceph-mon[28081]: pgmap v573: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:51.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:50 smithi161 ceph-mon[62387]: pgmap v574: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:51.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:50 smithi179 ceph-mon[63029]: pgmap v574: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:51.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:50 smithi084 ceph-mon[28081]: pgmap v574: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:56:53.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:52 smithi161 ceph-mon[62387]: pgmap v575: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:53.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:52 smithi179 ceph-mon[63029]: pgmap v575: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:53.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:52 smithi084 ceph-mon[28081]: pgmap v575: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:56:55.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:54 smithi161 ceph-mon[62387]: pgmap v576: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:55.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:54 smithi179 ceph-mon[63029]: pgmap v576: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:55.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:54 smithi084 ceph-mon[28081]: pgmap v576: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:57.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:56 smithi161 ceph-mon[62387]: pgmap v577: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:57.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:56 smithi179 ceph-mon[63029]: pgmap v577: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:57.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:56 smithi084 ceph-mon[28081]: pgmap v577: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:56:59.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:56:58 smithi161 ceph-mon[62387]: pgmap v578: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:59.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:56:58 smithi179 ceph-mon[63029]: pgmap v578: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:56:59.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:56:58 smithi084 ceph-mon[28081]: pgmap v578: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:01.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:00 smithi161 ceph-mon[62387]: pgmap v579: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:01.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:00 smithi179 ceph-mon[63029]: pgmap v579: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:01.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:00 smithi084 ceph-mon[28081]: pgmap v579: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:03.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:02 smithi161 ceph-mon[62387]: pgmap v580: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:03.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:02 smithi179 ceph-mon[63029]: pgmap v580: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:03.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:02 smithi084 ceph-mon[28081]: pgmap v580: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:05.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:04 smithi161 ceph-mon[62387]: pgmap v581: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:05.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:04 smithi179 ceph-mon[63029]: pgmap v581: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:05.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:04 smithi084 ceph-mon[28081]: pgmap v581: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:07.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:06 smithi161 ceph-mon[62387]: pgmap v582: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:07.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:06 smithi179 ceph-mon[63029]: pgmap v582: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:07.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:06 smithi084 ceph-mon[28081]: pgmap v582: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:09.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:08 smithi161 ceph-mon[62387]: pgmap v583: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:09.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:08 smithi179 ceph-mon[63029]: pgmap v583: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:09.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:08 smithi084 ceph-mon[28081]: pgmap v583: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:11.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:10 smithi161 ceph-mon[62387]: pgmap v584: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:11.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:10 smithi179 ceph-mon[63029]: pgmap v584: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:11.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:10 smithi084 ceph-mon[28081]: pgmap v584: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:13.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:12 smithi161 ceph-mon[62387]: pgmap v585: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:13.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:12 smithi179 ceph-mon[63029]: pgmap v585: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:13.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:12 smithi084 ceph-mon[28081]: pgmap v585: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:15.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:15 smithi179 ceph-mon[63029]: pgmap v586: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:15.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:15 smithi084 ceph-mon[28081]: pgmap v586: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:15.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:15 smithi161 ceph-mon[62387]: pgmap v586: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:17.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:17 smithi179 ceph-mon[63029]: pgmap v587: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:17.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:17 smithi084 ceph-mon[28081]: pgmap v587: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:17.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:17 smithi161 ceph-mon[62387]: pgmap v587: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:19.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:19 smithi179 ceph-mon[63029]: pgmap v588: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:19.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:19 smithi084 ceph-mon[28081]: pgmap v588: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:19.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:19 smithi161 ceph-mon[62387]: pgmap v588: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:21.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:21 smithi179 ceph-mon[63029]: pgmap v589: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:21.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:21 smithi084 ceph-mon[28081]: pgmap v589: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:21.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:21 smithi161 ceph-mon[62387]: pgmap v589: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:23.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:23 smithi179 ceph-mon[63029]: pgmap v590: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:23.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:23 smithi084 ceph-mon[28081]: pgmap v590: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:23.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:23 smithi161 ceph-mon[62387]: pgmap v590: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:25.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:25 smithi179 ceph-mon[63029]: pgmap v591: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:25.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:25 smithi084 ceph-mon[28081]: pgmap v591: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:25.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:25 smithi161 ceph-mon[62387]: pgmap v591: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:27.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:27 smithi179 ceph-mon[63029]: pgmap v592: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:27.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:27 smithi084 ceph-mon[28081]: pgmap v592: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:27.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:27 smithi161 ceph-mon[62387]: pgmap v592: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:29.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:29 smithi179 ceph-mon[63029]: pgmap v593: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:29.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:29 smithi084 ceph-mon[28081]: pgmap v593: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:29.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:29 smithi161 ceph-mon[62387]: pgmap v593: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:31.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:31 smithi179 ceph-mon[63029]: pgmap v594: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:31.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:31 smithi084 ceph-mon[28081]: pgmap v594: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:31.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:31 smithi161 ceph-mon[62387]: pgmap v594: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:33.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:33 smithi179 ceph-mon[63029]: pgmap v595: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:33.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:33 smithi084 ceph-mon[28081]: pgmap v595: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:33.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:33 smithi161 ceph-mon[62387]: pgmap v595: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:35.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:35 smithi179 ceph-mon[63029]: pgmap v596: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:35.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:35 smithi084 ceph-mon[28081]: pgmap v596: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:35.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:35 smithi161 ceph-mon[62387]: pgmap v596: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:37.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:57:37.046+0000 7f08cf400640 1 -- 172.21.15.84:0/2794133779 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f08b0005850 con 0x7f08d810ed90 2024-04-28T12:57:37.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:57:37.046+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 <== mon.0 v2:172.21.15.84:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f08b0005850 con 0x7f08d810ed90 2024-04-28T12:57:37.417 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:37 smithi179 ceph-mon[63029]: pgmap v597: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:37.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:37 smithi084 ceph-mon[28081]: pgmap v597: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:37.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:37 smithi161 ceph-mon[62387]: pgmap v597: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:39.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:39 smithi084 ceph-mon[28081]: pgmap v598: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:39.421 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:39 smithi179 ceph-mon[63029]: pgmap v598: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:39.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:39 smithi161 ceph-mon[62387]: pgmap v598: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:41.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:41 smithi179 ceph-mon[63029]: pgmap v599: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:41.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:41 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:57:41.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:41 smithi084 ceph-mon[28081]: pgmap v599: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:41.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:41 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:57:41.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:41 smithi161 ceph-mon[62387]: pgmap v599: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:41.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:41 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:57:42.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi179 to 176.2M 2024-04-28T12:57:42.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi161 to 176.2M 2024-04-28T12:57:42.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: Unable to set osd_memory_target on smithi179 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-28T12:57:42.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: Unable to set osd_memory_target on smithi161 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-28T12:57:42.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:57:42.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:57:42.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:57:42.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi179 to 176.2M 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi161 to 176.2M 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: Unable to set osd_memory_target on smithi179 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: Unable to set osd_memory_target on smithi161 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:57:42.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:57:42.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi179 to 176.2M 2024-04-28T12:57:42.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi161 to 176.2M 2024-04-28T12:57:42.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: Unable to set osd_memory_target on smithi179 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-28T12:57:42.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: Unable to set osd_memory_target on smithi161 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-28T12:57:42.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.747 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.747 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.747 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-28T12:57:42.747 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:57:42.747 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:57:42.747 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:57:43.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:43 smithi179 ceph-mon[63029]: pgmap v600: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:43.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:43 smithi179 ceph-mon[63029]: Adjusting osd_memory_target on smithi084 to 176.4M 2024-04-28T12:57:43.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:43 smithi179 ceph-mon[63029]: Unable to set osd_memory_target on smithi084 to 185061990: error parsing value: Value '185061990' is below minimum 939524096 2024-04-28T12:57:43.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:43 smithi084 ceph-mon[28081]: pgmap v600: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:43.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:43 smithi084 ceph-mon[28081]: Adjusting osd_memory_target on smithi084 to 176.4M 2024-04-28T12:57:43.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:43 smithi084 ceph-mon[28081]: Unable to set osd_memory_target on smithi084 to 185061990: error parsing value: Value '185061990' is below minimum 939524096 2024-04-28T12:57:43.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:43 smithi161 ceph-mon[62387]: pgmap v600: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:43.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:43 smithi161 ceph-mon[62387]: Adjusting osd_memory_target on smithi084 to 176.4M 2024-04-28T12:57:43.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:43 smithi161 ceph-mon[62387]: Unable to set osd_memory_target on smithi084 to 185061990: error parsing value: Value '185061990' is below minimum 939524096 2024-04-28T12:57:45.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:45 smithi179 ceph-mon[63029]: pgmap v601: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:45.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:45 smithi084 ceph-mon[28081]: pgmap v601: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:45.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:45 smithi161 ceph-mon[62387]: pgmap v601: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:47.667 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:47 smithi179 ceph-mon[63029]: pgmap v602: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:47.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:47 smithi084 ceph-mon[28081]: pgmap v602: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:47.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:47 smithi161 ceph-mon[62387]: pgmap v602: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:49.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:49 smithi179 ceph-mon[63029]: pgmap v603: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:49.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:49 smithi084 ceph-mon[28081]: pgmap v603: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:49.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:49 smithi161 ceph-mon[62387]: pgmap v603: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:51.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:51 smithi179 ceph-mon[63029]: pgmap v604: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:51.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:51 smithi084 ceph-mon[28081]: pgmap v604: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:51.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:51 smithi161 ceph-mon[62387]: pgmap v604: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:57:53.667 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:53 smithi179 ceph-mon[63029]: pgmap v605: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:53.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:53 smithi084 ceph-mon[28081]: pgmap v605: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:53.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:53 smithi161 ceph-mon[62387]: pgmap v605: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:57:55.669 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:55 smithi084 ceph-mon[28081]: pgmap v606: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:55.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:55 smithi161 ceph-mon[62387]: pgmap v606: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:55.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:55 smithi179 ceph-mon[63029]: pgmap v606: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:57.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:57 smithi161 ceph-mon[62387]: pgmap v607: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:57.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:57 smithi179 ceph-mon[63029]: pgmap v607: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:57.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:57 smithi084 ceph-mon[28081]: pgmap v607: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:57:59.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:57:59 smithi161 ceph-mon[62387]: pgmap v608: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:59.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:57:59 smithi179 ceph-mon[63029]: pgmap v608: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:57:59.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:57:59 smithi084 ceph-mon[28081]: pgmap v608: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:01.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:01 smithi161 ceph-mon[62387]: pgmap v609: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:01.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:01 smithi179 ceph-mon[63029]: pgmap v609: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:01.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:01 smithi084 ceph-mon[28081]: pgmap v609: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:03.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:03 smithi161 ceph-mon[62387]: pgmap v610: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:03.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:03 smithi179 ceph-mon[63029]: pgmap v610: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:03.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:03 smithi084 ceph-mon[28081]: pgmap v610: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:05.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:05 smithi161 ceph-mon[62387]: pgmap v611: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:05.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:05 smithi179 ceph-mon[63029]: pgmap v611: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:05.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:05 smithi084 ceph-mon[28081]: pgmap v611: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:07.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:07 smithi161 ceph-mon[62387]: pgmap v612: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:07.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:07 smithi179 ceph-mon[63029]: pgmap v612: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:07.922 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:07 smithi084 ceph-mon[28081]: pgmap v612: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:09.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:09 smithi161 ceph-mon[62387]: pgmap v613: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:09.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:09 smithi179 ceph-mon[63029]: pgmap v613: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:09.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:09 smithi084 ceph-mon[28081]: pgmap v613: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:11.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:11 smithi161 ceph-mon[62387]: pgmap v614: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:11.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:11 smithi179 ceph-mon[63029]: pgmap v614: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:11.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:11 smithi084 ceph-mon[28081]: pgmap v614: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:13.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:13 smithi161 ceph-mon[62387]: pgmap v615: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:13.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:13 smithi179 ceph-mon[63029]: pgmap v615: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:13.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:13 smithi084 ceph-mon[28081]: pgmap v615: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:15.723 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:15 smithi084 ceph-mon[28081]: pgmap v616: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:15.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:15 smithi161 ceph-mon[62387]: pgmap v616: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:15.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:15 smithi179 ceph-mon[63029]: pgmap v616: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:17.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:17 smithi161 ceph-mon[62387]: pgmap v617: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:17.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:17 smithi179 ceph-mon[63029]: pgmap v617: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:17.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:17 smithi084 ceph-mon[28081]: pgmap v617: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:19.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:19 smithi161 ceph-mon[62387]: pgmap v618: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:19.917 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:19 smithi179 ceph-mon[63029]: pgmap v618: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:19.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:19 smithi084 ceph-mon[28081]: pgmap v618: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:21.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:21 smithi161 ceph-mon[62387]: pgmap v619: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:21.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:21 smithi179 ceph-mon[63029]: pgmap v619: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:21.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:21 smithi084 ceph-mon[28081]: pgmap v619: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:23.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:23 smithi161 ceph-mon[62387]: pgmap v620: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:23.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:23 smithi179 ceph-mon[63029]: pgmap v620: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:23.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:23 smithi084 ceph-mon[28081]: pgmap v620: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:25.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:25 smithi161 ceph-mon[62387]: pgmap v621: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:25.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:25 smithi179 ceph-mon[63029]: pgmap v621: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:25.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:25 smithi084 ceph-mon[28081]: pgmap v621: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:27.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:27 smithi179 ceph-mon[63029]: pgmap v622: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:27.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:27 smithi084 ceph-mon[28081]: pgmap v622: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:27.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:27 smithi161 ceph-mon[62387]: pgmap v622: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:29.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:29 smithi179 ceph-mon[63029]: pgmap v623: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:29.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:29 smithi084 ceph-mon[28081]: pgmap v623: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:29.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:29 smithi161 ceph-mon[62387]: pgmap v623: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:31.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:31 smithi179 ceph-mon[63029]: pgmap v624: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:31.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:31 smithi084 ceph-mon[28081]: pgmap v624: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:31.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:31 smithi161 ceph-mon[62387]: pgmap v624: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:33.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:33 smithi179 ceph-mon[63029]: pgmap v625: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:33.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:33 smithi084 ceph-mon[28081]: pgmap v625: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:33.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:33 smithi161 ceph-mon[62387]: pgmap v625: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:35.798 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:35 smithi161 ceph-mon[62387]: pgmap v626: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:35.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:35 smithi179 ceph-mon[63029]: pgmap v626: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:35.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:35 smithi084 ceph-mon[28081]: pgmap v626: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:37.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:37 smithi179 ceph-mon[63029]: pgmap v627: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:37.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:37 smithi084 ceph-mon[28081]: pgmap v627: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:37.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:37 smithi161 ceph-mon[62387]: pgmap v627: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:39.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:39 smithi179 ceph-mon[63029]: pgmap v628: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:39.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:39 smithi084 ceph-mon[28081]: pgmap v628: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:39.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:39 smithi161 ceph-mon[62387]: pgmap v628: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:41.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:41 smithi179 ceph-mon[63029]: pgmap v629: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:41.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:41 smithi084 ceph-mon[28081]: pgmap v629: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:41.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:41 smithi161 ceph-mon[62387]: pgmap v629: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:42.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:42 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:58:42.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:42 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:58:42.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:42 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:58:43.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:43 smithi179 ceph-mon[63029]: pgmap v630: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:43.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:43 smithi084 ceph-mon[28081]: pgmap v630: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:43.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:43 smithi161 ceph-mon[62387]: pgmap v630: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:44.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:58:44.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:58:44.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:58:44.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:58:44.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:58:44.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:58:44.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:58:44.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:58:44.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:58:45.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:45 smithi161 ceph-mon[62387]: pgmap v631: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:46.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:45 smithi179 ceph-mon[63029]: pgmap v631: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:46.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:45 smithi084 ceph-mon[28081]: pgmap v631: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:47.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:47 smithi161 ceph-mon[62387]: pgmap v632: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:48.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:47 smithi179 ceph-mon[63029]: pgmap v632: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:48.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:47 smithi084 ceph-mon[28081]: pgmap v632: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:49.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:49 smithi161 ceph-mon[62387]: pgmap v633: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:50.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:49 smithi179 ceph-mon[63029]: pgmap v633: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:50.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:49 smithi084 ceph-mon[28081]: pgmap v633: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:52.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:51 smithi179 ceph-mon[63029]: pgmap v634: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:52.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:51 smithi084 ceph-mon[28081]: pgmap v634: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:52.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:51 smithi161 ceph-mon[62387]: pgmap v634: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:58:54.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:53 smithi179 ceph-mon[63029]: pgmap v635: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:54.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:53 smithi084 ceph-mon[28081]: pgmap v635: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:54.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:53 smithi161 ceph-mon[62387]: pgmap v635: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:58:56.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:55 smithi179 ceph-mon[63029]: pgmap v636: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:56.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:55 smithi084 ceph-mon[28081]: pgmap v636: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:56.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:55 smithi161 ceph-mon[62387]: pgmap v636: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:58:58.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:57 smithi179 ceph-mon[63029]: pgmap v637: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:58.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:57 smithi084 ceph-mon[28081]: pgmap v637: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:58:58.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:57 smithi161 ceph-mon[62387]: pgmap v637: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:00.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:58:59 smithi179 ceph-mon[63029]: pgmap v638: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:00.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:58:59 smithi084 ceph-mon[28081]: pgmap v638: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:00.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:58:59 smithi161 ceph-mon[62387]: pgmap v638: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:02.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:01 smithi179 ceph-mon[63029]: pgmap v639: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:02.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:01 smithi084 ceph-mon[28081]: pgmap v639: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:02.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:01 smithi161 ceph-mon[62387]: pgmap v639: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:04.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:03 smithi179 ceph-mon[63029]: pgmap v640: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:04.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:03 smithi084 ceph-mon[28081]: pgmap v640: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:04.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:03 smithi161 ceph-mon[62387]: pgmap v640: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:06.048 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:05 smithi161 ceph-mon[62387]: pgmap v641: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:06.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:05 smithi179 ceph-mon[63029]: pgmap v641: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:06.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:05 smithi084 ceph-mon[28081]: pgmap v641: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:08.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:07 smithi179 ceph-mon[63029]: pgmap v642: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-28T12:59:08.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:07 smithi084 ceph-mon[28081]: pgmap v642: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-28T12:59:08.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:07 smithi161 ceph-mon[62387]: pgmap v642: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-28T12:59:09.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:08 smithi179 ceph-mon[63029]: pgmap v643: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-28T12:59:09.169 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:08 smithi084 ceph-mon[28081]: pgmap v643: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-28T12:59:09.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:08 smithi161 ceph-mon[62387]: pgmap v643: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-28T12:59:11.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:10 smithi161 ceph-mon[62387]: pgmap v644: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:59:11.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:10 smithi179 ceph-mon[63029]: pgmap v644: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:59:11.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:10 smithi084 ceph-mon[28081]: pgmap v644: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T12:59:13.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:12 smithi161 ceph-mon[62387]: pgmap v645: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T12:59:13.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:12 smithi179 ceph-mon[63029]: pgmap v645: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T12:59:13.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:12 smithi084 ceph-mon[28081]: pgmap v645: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-28T12:59:15.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:14 smithi161 ceph-mon[62387]: pgmap v646: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-28T12:59:15.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:14 smithi179 ceph-mon[63029]: pgmap v646: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-28T12:59:15.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:14 smithi084 ceph-mon[28081]: pgmap v646: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-28T12:59:17.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:16 smithi161 ceph-mon[62387]: pgmap v647: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-28T12:59:17.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:16 smithi179 ceph-mon[63029]: pgmap v647: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-28T12:59:17.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:16 smithi084 ceph-mon[28081]: pgmap v647: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-28T12:59:19.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:18 smithi179 ceph-mon[63029]: pgmap v648: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:19.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:18 smithi084 ceph-mon[28081]: pgmap v648: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:19.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:18 smithi161 ceph-mon[62387]: pgmap v648: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:21.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:21 smithi179 ceph-mon[63029]: pgmap v649: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:21.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:21 smithi084 ceph-mon[28081]: pgmap v649: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:21.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:21 smithi161 ceph-mon[62387]: pgmap v649: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:23.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:23 smithi179 ceph-mon[63029]: pgmap v650: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:23.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:23 smithi084 ceph-mon[28081]: pgmap v650: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:23.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:23 smithi161 ceph-mon[62387]: pgmap v650: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:25.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:25 smithi179 ceph-mon[63029]: pgmap v651: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:25.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:25 smithi084 ceph-mon[28081]: pgmap v651: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:25.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:25 smithi161 ceph-mon[62387]: pgmap v651: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:27.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:27 smithi179 ceph-mon[63029]: pgmap v652: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:27.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:27 smithi084 ceph-mon[28081]: pgmap v652: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:27.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:27 smithi161 ceph-mon[62387]: pgmap v652: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:29.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:29 smithi161 ceph-mon[62387]: pgmap v653: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:29.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:29 smithi179 ceph-mon[63029]: pgmap v653: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:29.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:29 smithi084 ceph-mon[28081]: pgmap v653: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:31.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:31 smithi161 ceph-mon[62387]: pgmap v654: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:31.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:31 smithi179 ceph-mon[63029]: pgmap v654: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:31.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:31 smithi084 ceph-mon[28081]: pgmap v654: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-28T12:59:33.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:33 smithi161 ceph-mon[62387]: pgmap v655: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:33.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:33 smithi179 ceph-mon[63029]: pgmap v655: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:33.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:33 smithi084 ceph-mon[28081]: pgmap v655: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:35.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:35 smithi161 ceph-mon[62387]: pgmap v656: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:35.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:35 smithi179 ceph-mon[63029]: pgmap v656: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:35.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:35 smithi084 ceph-mon[28081]: pgmap v656: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:37.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:37 smithi161 ceph-mon[62387]: pgmap v657: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:37.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:37 smithi179 ceph-mon[63029]: pgmap v657: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:37.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:37 smithi084 ceph-mon[28081]: pgmap v657: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:39.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:39 smithi161 ceph-mon[62387]: pgmap v658: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:39.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:39 smithi179 ceph-mon[63029]: pgmap v658: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:39.922 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:39 smithi084 ceph-mon[28081]: pgmap v658: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:41.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:41 smithi161 ceph-mon[62387]: pgmap v659: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:41.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:41 smithi179 ceph-mon[63029]: pgmap v659: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:41.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:41 smithi084 ceph-mon[28081]: pgmap v659: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:43.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:43 smithi161 ceph-mon[62387]: pgmap v660: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:43.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:43 smithi179 ceph-mon[63029]: pgmap v660: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:43.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:43 smithi084 ceph-mon[28081]: pgmap v660: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:44.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:44 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:59:44.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:44 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:59:44.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:44 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T12:59:45.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:45 smithi161 ceph-mon[62387]: pgmap v661: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:45.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:45 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:59:45.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:45 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:59:45.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:45 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:59:45.822 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:45 smithi084 ceph-mon[28081]: pgmap v661: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:45.822 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:45 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:59:45.822 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:45 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:59:45.822 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:45 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:59:45.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:45 smithi179 ceph-mon[63029]: pgmap v661: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:45.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:45 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T12:59:45.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:45 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T12:59:45.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:45 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T12:59:47.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:47 smithi161 ceph-mon[62387]: pgmap v662: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:47.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:47 smithi179 ceph-mon[63029]: pgmap v662: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:47.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:47 smithi084 ceph-mon[28081]: pgmap v662: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:49.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:49 smithi161 ceph-mon[62387]: pgmap v663: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:49.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:49 smithi179 ceph-mon[63029]: pgmap v663: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:49.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:49 smithi084 ceph-mon[28081]: pgmap v663: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:51.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:51 smithi161 ceph-mon[62387]: pgmap v664: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:51.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:51 smithi179 ceph-mon[63029]: pgmap v664: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:51.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:51 smithi084 ceph-mon[28081]: pgmap v664: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T12:59:52.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:59:52.068+0000 7f08cf400640 1 -- 172.21.15.84:0/2794133779 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f08b0005b50 con 0x7f08d810ed90 2024-04-28T12:59:52.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T12:59:52.069+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 <== mon.0 v2:172.21.15.84:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f08b0005b50 con 0x7f08d810ed90 2024-04-28T12:59:53.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:53 smithi161 ceph-mon[62387]: pgmap v665: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:53.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:53 smithi179 ceph-mon[63029]: pgmap v665: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:53.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:53 smithi084 ceph-mon[28081]: pgmap v665: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T12:59:55.745 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:55 smithi161 ceph-mon[62387]: pgmap v666: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:55.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:55 smithi179 ceph-mon[63029]: pgmap v666: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:55.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:55 smithi084 ceph-mon[28081]: pgmap v666: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:57.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:57 smithi179 ceph-mon[63029]: pgmap v667: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:57.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:57 smithi084 ceph-mon[28081]: pgmap v667: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:57.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:57 smithi161 ceph-mon[62387]: pgmap v667: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T12:59:59.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 12:59:59 smithi179 ceph-mon[63029]: pgmap v668: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:59.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 12:59:59 smithi084 ceph-mon[28081]: pgmap v668: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T12:59:59.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 12:59:59 smithi161 ceph-mon[62387]: pgmap v668: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:00.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:00 smithi179 ceph-mon[63029]: overall HEALTH_OK 2024-04-28T13:00:00.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:00 smithi084 ceph-mon[28081]: overall HEALTH_OK 2024-04-28T13:00:00.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:00 smithi161 ceph-mon[62387]: overall HEALTH_OK 2024-04-28T13:00:01.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:01 smithi179 ceph-mon[63029]: pgmap v669: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:01.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:01 smithi084 ceph-mon[28081]: pgmap v669: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:01.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:01 smithi161 ceph-mon[62387]: pgmap v669: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:03.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:03 smithi179 ceph-mon[63029]: pgmap v670: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:03.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:03 smithi084 ceph-mon[28081]: pgmap v670: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:03.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:03 smithi161 ceph-mon[62387]: pgmap v670: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:05.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:05 smithi179 ceph-mon[63029]: pgmap v671: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:05.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:05 smithi084 ceph-mon[28081]: pgmap v671: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:05.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:05 smithi161 ceph-mon[62387]: pgmap v671: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:07.798 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:07 smithi161 ceph-mon[62387]: pgmap v672: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:07.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:07 smithi179 ceph-mon[63029]: pgmap v672: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:07.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:07 smithi084 ceph-mon[28081]: pgmap v672: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:09.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:09 smithi179 ceph-mon[63029]: pgmap v673: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:09.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:09 smithi084 ceph-mon[28081]: pgmap v673: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:09.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:09 smithi161 ceph-mon[62387]: pgmap v673: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:11.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:11 smithi179 ceph-mon[63029]: pgmap v674: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:11.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:11 smithi084 ceph-mon[28081]: pgmap v674: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:11.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:11 smithi161 ceph-mon[62387]: pgmap v674: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:13.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:13 smithi179 ceph-mon[63029]: pgmap v675: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:13.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:13 smithi084 ceph-mon[28081]: pgmap v675: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:13.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:13 smithi161 ceph-mon[62387]: pgmap v675: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:15.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:15 smithi179 ceph-mon[63029]: pgmap v676: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:15.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:15 smithi084 ceph-mon[28081]: pgmap v676: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:15.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:15 smithi161 ceph-mon[62387]: pgmap v676: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:17.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:17 smithi179 ceph-mon[63029]: pgmap v677: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:17.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:17 smithi084 ceph-mon[28081]: pgmap v677: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:17.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:17 smithi161 ceph-mon[62387]: pgmap v677: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:19.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:19 smithi179 ceph-mon[63029]: pgmap v678: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:19.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:19 smithi084 ceph-mon[28081]: pgmap v678: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:19.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:19 smithi161 ceph-mon[62387]: pgmap v678: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:21.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:21 smithi179 ceph-mon[63029]: pgmap v679: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:21.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:21 smithi084 ceph-mon[28081]: pgmap v679: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:21.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:21 smithi161 ceph-mon[62387]: pgmap v679: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:23.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:23 smithi179 ceph-mon[63029]: pgmap v680: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:23.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:23 smithi084 ceph-mon[28081]: pgmap v680: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:23.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:23 smithi161 ceph-mon[62387]: pgmap v680: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:25.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:25 smithi179 ceph-mon[63029]: pgmap v681: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:25.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:25 smithi084 ceph-mon[28081]: pgmap v681: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:25.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:25 smithi161 ceph-mon[62387]: pgmap v681: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:27.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:27 smithi179 ceph-mon[63029]: pgmap v682: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:27.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:27 smithi084 ceph-mon[28081]: pgmap v682: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:27.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:27 smithi161 ceph-mon[62387]: pgmap v682: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:29.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:29 smithi179 ceph-mon[63029]: pgmap v683: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:29.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:29 smithi084 ceph-mon[28081]: pgmap v683: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:29.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:29 smithi161 ceph-mon[62387]: pgmap v683: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:31.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:31 smithi179 ceph-mon[63029]: pgmap v684: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:31.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:31 smithi084 ceph-mon[28081]: pgmap v684: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:31.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:31 smithi161 ceph-mon[62387]: pgmap v684: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:33.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:33 smithi179 ceph-mon[63029]: pgmap v685: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:33.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:33 smithi084 ceph-mon[28081]: pgmap v685: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:33.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:33 smithi161 ceph-mon[62387]: pgmap v685: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:35.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:35 smithi179 ceph-mon[63029]: pgmap v686: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:35.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:35 smithi084 ceph-mon[28081]: pgmap v686: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:35.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:35 smithi161 ceph-mon[62387]: pgmap v686: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:37.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:00:37.076+0000 7f08cf400640 1 -- 172.21.15.84:0/2794133779 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f08b0005ea0 con 0x7f08d810ed90 2024-04-28T13:00:37.076 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:00:37.076+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 <== mon.0 v2:172.21.15.84:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f08b0005ea0 con 0x7f08d810ed90 2024-04-28T13:00:37.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:37 smithi179 ceph-mon[63029]: pgmap v687: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:37.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:37 smithi084 ceph-mon[28081]: pgmap v687: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:37.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:37 smithi161 ceph-mon[62387]: pgmap v687: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:39.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:39 smithi179 ceph-mon[63029]: pgmap v688: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:39.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:39 smithi084 ceph-mon[28081]: pgmap v688: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:39.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:39 smithi161 ceph-mon[62387]: pgmap v688: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:41.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:41 smithi179 ceph-mon[63029]: pgmap v689: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:41.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:41 smithi084 ceph-mon[28081]: pgmap v689: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:41.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:41 smithi161 ceph-mon[62387]: pgmap v689: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:43.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:43 smithi179 ceph-mon[63029]: pgmap v690: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:43.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:43 smithi084 ceph-mon[28081]: pgmap v690: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:43.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:43 smithi161 ceph-mon[62387]: pgmap v690: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:45.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:45 smithi179 ceph-mon[63029]: pgmap v691: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:45.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:45 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:00:45.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:45 smithi084 ceph-mon[28081]: pgmap v691: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:45.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:45 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:00:45.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:45 smithi161 ceph-mon[62387]: pgmap v691: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:45.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:45 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:00:47.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:47 smithi179 ceph-mon[63029]: pgmap v692: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:47.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:00:47.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:00:47.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:00:47.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:47 smithi084 ceph-mon[28081]: pgmap v692: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:47.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:00:47.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:00:47.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:00:47.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:47 smithi161 ceph-mon[62387]: pgmap v692: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:47.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:00:47.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:00:47.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:00:49.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:49 smithi179 ceph-mon[63029]: pgmap v693: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:49.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:49 smithi084 ceph-mon[28081]: pgmap v693: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:49.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:49 smithi161 ceph-mon[62387]: pgmap v693: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:51.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:51 smithi179 ceph-mon[63029]: pgmap v694: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:51.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:51 smithi084 ceph-mon[28081]: pgmap v694: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:51.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:51 smithi161 ceph-mon[62387]: pgmap v694: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:00:53.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:53 smithi179 ceph-mon[63029]: pgmap v695: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:53.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:53 smithi084 ceph-mon[28081]: pgmap v695: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:53.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:53 smithi161 ceph-mon[62387]: pgmap v695: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:00:55.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:55 smithi179 ceph-mon[63029]: pgmap v696: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:55.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:55 smithi084 ceph-mon[28081]: pgmap v696: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:55.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:55 smithi161 ceph-mon[62387]: pgmap v696: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:57.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:57 smithi179 ceph-mon[63029]: pgmap v697: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:57.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:57 smithi084 ceph-mon[28081]: pgmap v697: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:57.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:57 smithi161 ceph-mon[62387]: pgmap v697: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:00:59.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:00:59 smithi179 ceph-mon[63029]: pgmap v698: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:59.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:00:59 smithi084 ceph-mon[28081]: pgmap v698: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:00:59.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:00:59 smithi161 ceph-mon[62387]: pgmap v698: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:01.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:01 smithi161 ceph-mon[62387]: pgmap v699: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:01.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:01 smithi179 ceph-mon[63029]: pgmap v699: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:01.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:01 smithi084 ceph-mon[28081]: pgmap v699: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:03.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:03 smithi161 ceph-mon[62387]: pgmap v700: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:04.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:03 smithi179 ceph-mon[63029]: pgmap v700: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:04.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:03 smithi084 ceph-mon[28081]: pgmap v700: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:05.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:05 smithi161 ceph-mon[62387]: pgmap v701: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:06.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:05 smithi179 ceph-mon[63029]: pgmap v701: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:06.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:05 smithi084 ceph-mon[28081]: pgmap v701: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:07.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:07 smithi161 ceph-mon[62387]: pgmap v702: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:08.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:07 smithi179 ceph-mon[63029]: pgmap v702: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:08.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:07 smithi084 ceph-mon[28081]: pgmap v702: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:09.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:09 smithi161 ceph-mon[62387]: pgmap v703: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:10.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:09 smithi179 ceph-mon[63029]: pgmap v703: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:10.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:09 smithi084 ceph-mon[28081]: pgmap v703: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:11.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:11 smithi161 ceph-mon[62387]: pgmap v704: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:12.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:11 smithi179 ceph-mon[63029]: pgmap v704: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:12.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:11 smithi084 ceph-mon[28081]: pgmap v704: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:13.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:13 smithi161 ceph-mon[62387]: pgmap v705: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:14.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:13 smithi179 ceph-mon[63029]: pgmap v705: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:14.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:13 smithi084 ceph-mon[28081]: pgmap v705: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:15.973 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:15 smithi084 ceph-mon[28081]: pgmap v706: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:15.995 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:15 smithi161 ceph-mon[62387]: pgmap v706: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:16.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:15 smithi179 ceph-mon[63029]: pgmap v706: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:17.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:17 smithi161 ceph-mon[62387]: pgmap v707: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:18.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:17 smithi179 ceph-mon[63029]: pgmap v707: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:18.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:17 smithi084 ceph-mon[28081]: pgmap v707: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:19.970 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:19 smithi179 ceph-mon[63029]: pgmap v708: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:19.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:19 smithi161 ceph-mon[62387]: pgmap v708: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:20.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:19 smithi084 ceph-mon[28081]: pgmap v708: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:21.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:21 smithi161 ceph-mon[62387]: pgmap v709: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:22.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:21 smithi179 ceph-mon[63029]: pgmap v709: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:22.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:21 smithi084 ceph-mon[28081]: pgmap v709: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:23.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:23 smithi161 ceph-mon[62387]: pgmap v710: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:24.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:23 smithi179 ceph-mon[63029]: pgmap v710: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:24.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:23 smithi084 ceph-mon[28081]: pgmap v710: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:25.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:25 smithi161 ceph-mon[62387]: pgmap v711: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:26.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:25 smithi179 ceph-mon[63029]: pgmap v711: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:26.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:25 smithi084 ceph-mon[28081]: pgmap v711: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:27.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:27 smithi161 ceph-mon[62387]: pgmap v712: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:28.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:27 smithi179 ceph-mon[63029]: pgmap v712: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:28.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:27 smithi084 ceph-mon[28081]: pgmap v712: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:29.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:29 smithi161 ceph-mon[62387]: pgmap v713: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:30.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:29 smithi179 ceph-mon[63029]: pgmap v713: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:30.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:29 smithi084 ceph-mon[28081]: pgmap v713: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:32.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:31 smithi179 ceph-mon[63029]: pgmap v714: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:32.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:31 smithi084 ceph-mon[28081]: pgmap v714: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:32.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:31 smithi161 ceph-mon[62387]: pgmap v714: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:34.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:33 smithi179 ceph-mon[63029]: pgmap v715: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:34.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:33 smithi084 ceph-mon[28081]: pgmap v715: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:34.245 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:33 smithi161 ceph-mon[62387]: pgmap v715: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:36.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:35 smithi179 ceph-mon[63029]: pgmap v716: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:36.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:35 smithi084 ceph-mon[28081]: pgmap v716: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:36.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:35 smithi161 ceph-mon[62387]: pgmap v716: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:38.048 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:37 smithi161 ceph-mon[62387]: pgmap v717: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:38.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:37 smithi179 ceph-mon[63029]: pgmap v717: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:38.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:37 smithi084 ceph-mon[28081]: pgmap v717: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:40.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:39 smithi179 ceph-mon[63029]: pgmap v718: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:40.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:39 smithi084 ceph-mon[28081]: pgmap v718: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:40.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:39 smithi161 ceph-mon[62387]: pgmap v718: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:42.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:41 smithi179 ceph-mon[63029]: pgmap v719: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:42.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:41 smithi084 ceph-mon[28081]: pgmap v719: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:42.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:41 smithi161 ceph-mon[62387]: pgmap v719: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:44.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:43 smithi179 ceph-mon[63029]: pgmap v720: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:44.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:43 smithi084 ceph-mon[28081]: pgmap v720: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:44.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:43 smithi161 ceph-mon[62387]: pgmap v720: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:46.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:45 smithi179 ceph-mon[63029]: pgmap v721: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:46.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:45 smithi084 ceph-mon[28081]: pgmap v721: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:46.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:45 smithi161 ceph-mon[62387]: pgmap v721: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:48.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:47 smithi179 ceph-mon[63029]: pgmap v722: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:48.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:01:48.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:48.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:48.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:48.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:47 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:48.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:47 smithi084 ceph-mon[28081]: pgmap v722: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:48.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:01:48.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:48.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:48.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:48.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:47 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:48.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:47 smithi161 ceph-mon[62387]: pgmap v722: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:48.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:01:48.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:48.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:48.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:48.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:47 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:49.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:49 smithi084 ceph-mon[28081]: pgmap v723: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:49.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:49.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:49 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:49.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:49 smithi179 ceph-mon[63029]: pgmap v723: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:49.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:49.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:49 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:49.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:49 smithi161 ceph-mon[62387]: pgmap v723: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:49.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:49.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:49 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:50.631 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:50.631 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:50.632 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:50.632 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:50 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:50.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:50.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:50.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:50.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:50 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:50.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:50.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:50.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:50.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:50 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:51.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:51 smithi179 ceph-mon[63029]: pgmap v724: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:51.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:51 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:51.668 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:51 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:51.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:51 smithi084 ceph-mon[28081]: pgmap v724: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:51.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:51 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:51.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:51 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:51.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:51 smithi161 ceph-mon[62387]: pgmap v724: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:01:51.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:51 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:51.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:51 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:52.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:01:52.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:01:52.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:52.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:01:52.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:01:52.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:52.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:01:52.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:01:52.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:01:53.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:53 smithi161 ceph-mon[62387]: pgmap v725: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:53.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:53 smithi179 ceph-mon[63029]: pgmap v725: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:53.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:53 smithi084 ceph-mon[28081]: pgmap v725: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:01:55.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:55 smithi161 ceph-mon[62387]: pgmap v726: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:55.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:55 smithi179 ceph-mon[63029]: pgmap v726: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:55.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:55 smithi084 ceph-mon[28081]: pgmap v726: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:57.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:57 smithi161 ceph-mon[62387]: pgmap v727: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:57.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:57 smithi179 ceph-mon[63029]: pgmap v727: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:57.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:57 smithi084 ceph-mon[28081]: pgmap v727: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:01:59.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:01:59 smithi161 ceph-mon[62387]: pgmap v728: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:59.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:01:59 smithi179 ceph-mon[63029]: pgmap v728: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:01:59.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:01:59 smithi084 ceph-mon[28081]: pgmap v728: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:01.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:01 smithi161 ceph-mon[62387]: pgmap v729: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:01.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:01 smithi179 ceph-mon[63029]: pgmap v729: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:01.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:01 smithi084 ceph-mon[28081]: pgmap v729: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:03.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:03 smithi161 ceph-mon[62387]: pgmap v730: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:03.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:03 smithi179 ceph-mon[63029]: pgmap v730: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:03.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:03 smithi084 ceph-mon[28081]: pgmap v730: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:05.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:05 smithi161 ceph-mon[62387]: pgmap v731: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:05.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:05 smithi179 ceph-mon[63029]: pgmap v731: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:05.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:05 smithi084 ceph-mon[28081]: pgmap v731: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:07.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:07 smithi161 ceph-mon[62387]: pgmap v732: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:07.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:07 smithi179 ceph-mon[63029]: pgmap v732: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:07.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:07 smithi084 ceph-mon[28081]: pgmap v732: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:09.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:09 smithi161 ceph-mon[62387]: pgmap v733: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:09.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:09 smithi179 ceph-mon[63029]: pgmap v733: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:09.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:09 smithi084 ceph-mon[28081]: pgmap v733: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:11.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:11 smithi161 ceph-mon[62387]: pgmap v734: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:11.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:11 smithi179 ceph-mon[63029]: pgmap v734: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:11.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:11 smithi084 ceph-mon[28081]: pgmap v734: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:13.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:13 smithi161 ceph-mon[62387]: pgmap v735: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:13.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:13 smithi179 ceph-mon[63029]: pgmap v735: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:13.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:13 smithi084 ceph-mon[28081]: pgmap v735: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:15.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:15 smithi161 ceph-mon[62387]: pgmap v736: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:15.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:15 smithi084 ceph-mon[28081]: pgmap v736: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:15.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:15 smithi179 ceph-mon[63029]: pgmap v736: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:17.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:17 smithi084 ceph-mon[28081]: pgmap v737: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:17.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:17 smithi161 ceph-mon[62387]: pgmap v737: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:17.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:17 smithi179 ceph-mon[63029]: pgmap v737: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:19.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:19 smithi161 ceph-mon[62387]: pgmap v738: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:19.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:19 smithi179 ceph-mon[63029]: pgmap v738: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:19.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:19 smithi084 ceph-mon[28081]: pgmap v738: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:21.746 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:21 smithi161 ceph-mon[62387]: pgmap v739: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:21.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:21 smithi179 ceph-mon[63029]: pgmap v739: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:21.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:21 smithi084 ceph-mon[28081]: pgmap v739: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:23.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:23 smithi179 ceph-mon[63029]: pgmap v740: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:23.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:23 smithi084 ceph-mon[28081]: pgmap v740: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:23.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:23 smithi161 ceph-mon[62387]: pgmap v740: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:25.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:25 smithi179 ceph-mon[63029]: pgmap v741: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:25.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:25 smithi084 ceph-mon[28081]: pgmap v741: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:25.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:25 smithi161 ceph-mon[62387]: pgmap v741: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:27.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:27 smithi179 ceph-mon[63029]: pgmap v742: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:27.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:27 smithi084 ceph-mon[28081]: pgmap v742: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:27.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:27 smithi161 ceph-mon[62387]: pgmap v742: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:29.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:29 smithi179 ceph-mon[63029]: pgmap v743: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:29.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:29 smithi084 ceph-mon[28081]: pgmap v743: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:29.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:29 smithi161 ceph-mon[62387]: pgmap v743: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:31.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:31 smithi179 ceph-mon[63029]: pgmap v744: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:31.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:31 smithi084 ceph-mon[28081]: pgmap v744: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:31.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:31 smithi161 ceph-mon[62387]: pgmap v744: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:33.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:33 smithi179 ceph-mon[63029]: pgmap v745: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:33.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:33 smithi084 ceph-mon[28081]: pgmap v745: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:33.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:33 smithi161 ceph-mon[62387]: pgmap v745: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:35.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:35 smithi179 ceph-mon[63029]: pgmap v746: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:35.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:35 smithi084 ceph-mon[28081]: pgmap v746: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:35.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:35 smithi161 ceph-mon[62387]: pgmap v746: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:37.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:37 smithi179 ceph-mon[63029]: pgmap v747: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:37.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:37 smithi084 ceph-mon[28081]: pgmap v747: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:37.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:37 smithi161 ceph-mon[62387]: pgmap v747: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:39.798 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:39 smithi161 ceph-mon[62387]: pgmap v748: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:39.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:39 smithi179 ceph-mon[63029]: pgmap v748: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:39.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:39 smithi084 ceph-mon[28081]: pgmap v748: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:41.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:41 smithi179 ceph-mon[63029]: pgmap v749: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:41.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:41 smithi084 ceph-mon[28081]: pgmap v749: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:41.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:41 smithi161 ceph-mon[62387]: pgmap v749: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:43.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:43 smithi179 ceph-mon[63029]: pgmap v750: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:43.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:43 smithi084 ceph-mon[28081]: pgmap v750: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:43.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:43 smithi161 ceph-mon[62387]: pgmap v750: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:45.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:45 smithi179 ceph-mon[63029]: pgmap v751: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:45.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:45 smithi084 ceph-mon[28081]: pgmap v751: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:45.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:45 smithi161 ceph-mon[62387]: pgmap v751: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:47.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:47 smithi179 ceph-mon[63029]: pgmap v752: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:47.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:47 smithi084 ceph-mon[28081]: pgmap v752: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:47.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:47 smithi161 ceph-mon[62387]: pgmap v752: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:49.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:49 smithi179 ceph-mon[63029]: pgmap v753: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:49.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:49 smithi084 ceph-mon[28081]: pgmap v753: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:49.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:49 smithi161 ceph-mon[62387]: pgmap v753: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:51.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:51 smithi179 ceph-mon[63029]: pgmap v754: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:51.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:51 smithi084 ceph-mon[28081]: pgmap v754: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:51.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:51 smithi161 ceph-mon[62387]: pgmap v754: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:02:52.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:52 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:02:52.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:52 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:02:52.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:52 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:02:53.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:53 smithi179 ceph-mon[63029]: pgmap v755: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:53.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:53 smithi084 ceph-mon[28081]: pgmap v755: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:53.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:53 smithi161 ceph-mon[62387]: pgmap v755: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:02:54.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:54 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:02:54.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:54 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:02:54.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:54 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:02:54.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:54 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:02:54.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:54 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:02:54.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:54 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:02:54.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:54 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:02:54.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:54 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:02:54.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:54 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:02:56.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:55 smithi179 ceph-mon[63029]: pgmap v756: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:56.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:55 smithi084 ceph-mon[28081]: pgmap v756: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:56.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:55 smithi161 ceph-mon[62387]: pgmap v756: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:02:58.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:57 smithi179 ceph-mon[63029]: pgmap v757: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:58.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:57 smithi084 ceph-mon[28081]: pgmap v757: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:02:58.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:57 smithi161 ceph-mon[62387]: pgmap v757: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:00.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:02:59 smithi179 ceph-mon[63029]: pgmap v758: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:00.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:02:59 smithi084 ceph-mon[28081]: pgmap v758: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:00.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:02:59 smithi161 ceph-mon[62387]: pgmap v758: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:02.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:01 smithi179 ceph-mon[63029]: pgmap v759: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:02.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:01 smithi084 ceph-mon[28081]: pgmap v759: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:02.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:01 smithi161 ceph-mon[62387]: pgmap v759: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:04.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:03 smithi179 ceph-mon[63029]: pgmap v760: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:04.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:03 smithi084 ceph-mon[28081]: pgmap v760: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:04.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:03 smithi161 ceph-mon[62387]: pgmap v760: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:05.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:04 smithi179 ceph-mon[63029]: pgmap v761: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:05.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:04 smithi084 ceph-mon[28081]: pgmap v761: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:05.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:04 smithi161 ceph-mon[62387]: pgmap v761: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:07.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:07 smithi179 ceph-mon[63029]: pgmap v762: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:07.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:07 smithi084 ceph-mon[28081]: pgmap v762: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:07.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:07 smithi161 ceph-mon[62387]: pgmap v762: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:09.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:09 smithi179 ceph-mon[63029]: pgmap v763: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:09.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:09 smithi084 ceph-mon[28081]: pgmap v763: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:09.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:09 smithi161 ceph-mon[62387]: pgmap v763: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:11.419 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:11 smithi179 ceph-mon[63029]: pgmap v764: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:11.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:11 smithi084 ceph-mon[28081]: pgmap v764: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:11.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:11 smithi161 ceph-mon[62387]: pgmap v764: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:13.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:13 smithi179 ceph-mon[63029]: pgmap v765: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:13.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:13 smithi084 ceph-mon[28081]: pgmap v765: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:13.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:13 smithi161 ceph-mon[62387]: pgmap v765: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:15.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:15 smithi179 ceph-mon[63029]: pgmap v766: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:15.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:15 smithi084 ceph-mon[28081]: pgmap v766: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:15.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:15 smithi161 ceph-mon[62387]: pgmap v766: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:17.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:17 smithi179 ceph-mon[63029]: pgmap v767: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:17.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:17 smithi084 ceph-mon[28081]: pgmap v767: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:17.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:17 smithi161 ceph-mon[62387]: pgmap v767: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:19.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:19 smithi179 ceph-mon[63029]: pgmap v768: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:19.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:19 smithi084 ceph-mon[28081]: pgmap v768: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:19.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:19 smithi161 ceph-mon[62387]: pgmap v768: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:21.419 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:21 smithi179 ceph-mon[63029]: pgmap v769: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:21.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:21 smithi084 ceph-mon[28081]: pgmap v769: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:21.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:21 smithi161 ceph-mon[62387]: pgmap v769: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:23.418 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:23 smithi179 ceph-mon[63029]: pgmap v770: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:23.420 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:23 smithi084 ceph-mon[28081]: pgmap v770: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:23.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:23 smithi161 ceph-mon[62387]: pgmap v770: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:25.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:25 smithi179 ceph-mon[63029]: pgmap v771: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:25.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:25 smithi084 ceph-mon[28081]: pgmap v771: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:25.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:25 smithi161 ceph-mon[62387]: pgmap v771: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:27.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:27 smithi179 ceph-mon[63029]: pgmap v772: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:27.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:27 smithi084 ceph-mon[28081]: pgmap v772: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:27.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:27 smithi161 ceph-mon[62387]: pgmap v772: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:29.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:29 smithi179 ceph-mon[63029]: pgmap v773: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:29.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:29 smithi084 ceph-mon[28081]: pgmap v773: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:29.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:29 smithi161 ceph-mon[62387]: pgmap v773: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:31.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:31 smithi179 ceph-mon[63029]: pgmap v774: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:31.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:31 smithi084 ceph-mon[28081]: pgmap v774: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:31.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:31 smithi161 ceph-mon[62387]: pgmap v774: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:33.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:33 smithi179 ceph-mon[63029]: pgmap v775: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:33.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:33 smithi084 ceph-mon[28081]: pgmap v775: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:33.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:33 smithi161 ceph-mon[62387]: pgmap v775: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:35.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:35 smithi179 ceph-mon[63029]: pgmap v776: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:35.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:35 smithi084 ceph-mon[28081]: pgmap v776: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:35.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:35 smithi161 ceph-mon[62387]: pgmap v776: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:37.106 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:03:37.106+0000 7f08cf400640 1 -- 172.21.15.84:0/2794133779 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f08b00060c0 con 0x7f08d810ed90 2024-04-28T13:03:37.106 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:03:37.107+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 <== mon.0 v2:172.21.15.84:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f08b00060c0 con 0x7f08d810ed90 2024-04-28T13:03:37.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:37 smithi179 ceph-mon[63029]: pgmap v777: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:37.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:37 smithi084 ceph-mon[28081]: pgmap v777: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:37.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:37 smithi161 ceph-mon[62387]: pgmap v777: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:39.799 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:39 smithi161 ceph-mon[62387]: pgmap v778: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:39 smithi179 ceph-mon[63029]: pgmap v778: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:39.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:39 smithi084 ceph-mon[28081]: pgmap v778: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:41.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:41 smithi179 ceph-mon[63029]: pgmap v779: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:41.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:41 smithi084 ceph-mon[28081]: pgmap v779: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:41.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:41 smithi161 ceph-mon[62387]: pgmap v779: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:43.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:43 smithi179 ceph-mon[63029]: pgmap v780: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:43.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:43 smithi084 ceph-mon[28081]: pgmap v780: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:43.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:43 smithi161 ceph-mon[62387]: pgmap v780: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:45.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:45 smithi179 ceph-mon[63029]: pgmap v781: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:45.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:45 smithi084 ceph-mon[28081]: pgmap v781: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:45.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:45 smithi161 ceph-mon[62387]: pgmap v781: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:47.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:47 smithi179 ceph-mon[63029]: pgmap v782: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:47.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:47 smithi084 ceph-mon[28081]: pgmap v782: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:47.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:47 smithi161 ceph-mon[62387]: pgmap v782: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:49.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:49 smithi179 ceph-mon[63029]: pgmap v783: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:49.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:49 smithi084 ceph-mon[28081]: pgmap v783: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:49.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:49 smithi161 ceph-mon[62387]: pgmap v783: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:51.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:51 smithi179 ceph-mon[63029]: pgmap v784: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:51.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:51 smithi084 ceph-mon[28081]: pgmap v784: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:51.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:51 smithi161 ceph-mon[62387]: pgmap v784: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:03:53.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:53 smithi179 ceph-mon[63029]: pgmap v785: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:53.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:53 smithi084 ceph-mon[28081]: pgmap v785: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:53.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:53 smithi161 ceph-mon[62387]: pgmap v785: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:03:54.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:54 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:03:54.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:54 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:03:54.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:54 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:03:55.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:55 smithi179 ceph-mon[63029]: pgmap v786: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:55.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:03:55.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:03:55.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:03:55.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:55 smithi084 ceph-mon[28081]: pgmap v786: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:55.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:03:55.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:03:55.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:03:55.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:55 smithi161 ceph-mon[62387]: pgmap v786: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:55.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:03:55.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:03:55.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:03:57.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:57 smithi179 ceph-mon[63029]: pgmap v787: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:57.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:57 smithi084 ceph-mon[28081]: pgmap v787: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:57.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:57 smithi161 ceph-mon[62387]: pgmap v787: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:03:59.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:03:59 smithi179 ceph-mon[63029]: pgmap v788: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:59.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:03:59 smithi084 ceph-mon[28081]: pgmap v788: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:03:59.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:03:59 smithi161 ceph-mon[62387]: pgmap v788: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:01.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:01 smithi179 ceph-mon[63029]: pgmap v789: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:01.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:01 smithi084 ceph-mon[28081]: pgmap v789: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:01.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:01 smithi161 ceph-mon[62387]: pgmap v789: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:03.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:03 smithi179 ceph-mon[63029]: pgmap v790: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:03.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:03 smithi084 ceph-mon[28081]: pgmap v790: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:03.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:03 smithi161 ceph-mon[62387]: pgmap v790: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:05.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:05 smithi179 ceph-mon[63029]: pgmap v791: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:05.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:05 smithi084 ceph-mon[28081]: pgmap v791: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:05.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:05 smithi161 ceph-mon[62387]: pgmap v791: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:07.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:07 smithi179 ceph-mon[63029]: pgmap v792: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:07.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:07 smithi084 ceph-mon[28081]: pgmap v792: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:07.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:07 smithi161 ceph-mon[62387]: pgmap v792: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:09.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:09 smithi179 ceph-mon[63029]: pgmap v793: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:09.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:09 smithi084 ceph-mon[28081]: pgmap v793: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:09.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:09 smithi161 ceph-mon[62387]: pgmap v793: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:11.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:11 smithi179 ceph-mon[63029]: pgmap v794: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:11.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:11 smithi084 ceph-mon[28081]: pgmap v794: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:11.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:11 smithi161 ceph-mon[62387]: pgmap v794: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:13.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:13 smithi179 ceph-mon[63029]: pgmap v795: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:13.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:13 smithi084 ceph-mon[28081]: pgmap v795: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:13.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:13 smithi161 ceph-mon[62387]: pgmap v795: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:15.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:15 smithi179 ceph-mon[63029]: pgmap v796: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:15.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:15 smithi084 ceph-mon[28081]: pgmap v796: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:15.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:15 smithi161 ceph-mon[62387]: pgmap v796: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:17.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:17 smithi179 ceph-mon[63029]: pgmap v797: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:17.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:17 smithi084 ceph-mon[28081]: pgmap v797: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:17.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:17 smithi161 ceph-mon[62387]: pgmap v797: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:19.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:19 smithi179 ceph-mon[63029]: pgmap v798: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:19.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:19 smithi084 ceph-mon[28081]: pgmap v798: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:19.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:19 smithi161 ceph-mon[62387]: pgmap v798: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:21.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:21 smithi179 ceph-mon[63029]: pgmap v799: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:21.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:21 smithi084 ceph-mon[28081]: pgmap v799: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:21.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:21 smithi161 ceph-mon[62387]: pgmap v799: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:23.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:23 smithi179 ceph-mon[63029]: pgmap v800: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:23.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:23 smithi084 ceph-mon[28081]: pgmap v800: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:23.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:23 smithi161 ceph-mon[62387]: pgmap v800: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:25.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:25 smithi179 ceph-mon[63029]: pgmap v801: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:25.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:25 smithi084 ceph-mon[28081]: pgmap v801: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:25.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:25 smithi161 ceph-mon[62387]: pgmap v801: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:27.918 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:27 smithi179 ceph-mon[63029]: pgmap v802: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:27.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:27 smithi084 ceph-mon[28081]: pgmap v802: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:27.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:27 smithi161 ceph-mon[62387]: pgmap v802: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:29.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:29 smithi161 ceph-mon[62387]: pgmap v803: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:30.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:29 smithi179 ceph-mon[63029]: pgmap v803: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:30.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:29 smithi084 ceph-mon[28081]: pgmap v803: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:31.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:31 smithi161 ceph-mon[62387]: pgmap v804: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:32.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:31 smithi179 ceph-mon[63029]: pgmap v804: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:32.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:31 smithi084 ceph-mon[28081]: pgmap v804: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:33.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:33 smithi161 ceph-mon[62387]: pgmap v805: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:34.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:33 smithi179 ceph-mon[63029]: pgmap v805: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:34.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:33 smithi084 ceph-mon[28081]: pgmap v805: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:35.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:35 smithi161 ceph-mon[62387]: pgmap v806: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:36.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:35 smithi179 ceph-mon[63029]: pgmap v806: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:36.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:35 smithi084 ceph-mon[28081]: pgmap v806: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:37.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:37 smithi161 ceph-mon[62387]: pgmap v807: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:38.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:37 smithi179 ceph-mon[63029]: pgmap v807: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:38.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:37 smithi084 ceph-mon[28081]: pgmap v807: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:39.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:39 smithi161 ceph-mon[62387]: pgmap v808: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:40.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:39 smithi179 ceph-mon[63029]: pgmap v808: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:40.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:39 smithi084 ceph-mon[28081]: pgmap v808: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:41.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:41 smithi161 ceph-mon[62387]: pgmap v809: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:42.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:41 smithi179 ceph-mon[63029]: pgmap v809: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:42.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:41 smithi084 ceph-mon[28081]: pgmap v809: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:43.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:43 smithi161 ceph-mon[62387]: pgmap v810: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:44.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:43 smithi179 ceph-mon[63029]: pgmap v810: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:44.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:43 smithi084 ceph-mon[28081]: pgmap v810: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:45.974 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:45 smithi084 ceph-mon[28081]: pgmap v811: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:45.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:45 smithi161 ceph-mon[62387]: pgmap v811: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:46.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:45 smithi179 ceph-mon[63029]: pgmap v811: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:47.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:47 smithi161 ceph-mon[62387]: pgmap v812: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:48.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:47 smithi179 ceph-mon[63029]: pgmap v812: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:48.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:47 smithi084 ceph-mon[28081]: pgmap v812: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:49.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:49 smithi161 ceph-mon[62387]: pgmap v813: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:50.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:49 smithi179 ceph-mon[63029]: pgmap v813: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:50.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:49 smithi084 ceph-mon[28081]: pgmap v813: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:51.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:51 smithi161 ceph-mon[62387]: pgmap v814: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:52.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:51 smithi179 ceph-mon[63029]: pgmap v814: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:52.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:51 smithi084 ceph-mon[28081]: pgmap v814: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:04:53.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:53 smithi161 ceph-mon[62387]: pgmap v815: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:54.168 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:53 smithi179 ceph-mon[63029]: pgmap v815: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:54.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:53 smithi084 ceph-mon[28081]: pgmap v815: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:04:55.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:55 smithi161 ceph-mon[62387]: pgmap v816: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:55.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:55 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:04:56.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:55 smithi179 ceph-mon[63029]: pgmap v816: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:56.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:55 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:04:56.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:55 smithi084 ceph-mon[28081]: pgmap v816: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:04:56.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:55 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:04:57.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:57 smithi161 ceph-mon[62387]: pgmap v817: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:57.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:57 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:04:57.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:57 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:04:57.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:57 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:04:58.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:57 smithi179 ceph-mon[63029]: pgmap v817: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:58.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:57 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:04:58.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:57 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:04:58.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:57 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:04:58.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:57 smithi084 ceph-mon[28081]: pgmap v817: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:04:58.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:57 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:04:58.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:57 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:04:58.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:57 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:05:00.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:04:59 smithi179 ceph-mon[63029]: pgmap v818: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:00.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:04:59 smithi084 ceph-mon[28081]: pgmap v818: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:00.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:04:59 smithi161 ceph-mon[62387]: pgmap v818: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:02.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:01 smithi179 ceph-mon[63029]: pgmap v819: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:02.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:01 smithi084 ceph-mon[28081]: pgmap v819: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:02.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:01 smithi161 ceph-mon[62387]: pgmap v819: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:04.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:03 smithi179 ceph-mon[63029]: pgmap v820: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:04.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:03 smithi084 ceph-mon[28081]: pgmap v820: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:04.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:03 smithi161 ceph-mon[62387]: pgmap v820: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:06.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:05 smithi179 ceph-mon[63029]: pgmap v821: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:06.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:05 smithi084 ceph-mon[28081]: pgmap v821: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:06.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:05 smithi161 ceph-mon[62387]: pgmap v821: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:08.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:07 smithi179 ceph-mon[63029]: pgmap v822: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:08.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:07 smithi084 ceph-mon[28081]: pgmap v822: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:08.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:07 smithi161 ceph-mon[62387]: pgmap v822: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:10.049 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:09 smithi161 ceph-mon[62387]: pgmap v823: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:10.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:09 smithi179 ceph-mon[63029]: pgmap v823: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:10.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:09 smithi084 ceph-mon[28081]: pgmap v823: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:12.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:11 smithi179 ceph-mon[63029]: pgmap v824: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:12.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:11 smithi084 ceph-mon[28081]: pgmap v824: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:12.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:11 smithi161 ceph-mon[62387]: pgmap v824: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:14.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:13 smithi179 ceph-mon[63029]: pgmap v825: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:14.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:13 smithi084 ceph-mon[28081]: pgmap v825: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:14.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:13 smithi161 ceph-mon[62387]: pgmap v825: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:16.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:15 smithi179 ceph-mon[63029]: pgmap v826: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:16.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:15 smithi084 ceph-mon[28081]: pgmap v826: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:16.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:15 smithi161 ceph-mon[62387]: pgmap v826: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:18.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:17 smithi179 ceph-mon[63029]: pgmap v827: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:18.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:17 smithi084 ceph-mon[28081]: pgmap v827: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:18.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:17 smithi161 ceph-mon[62387]: pgmap v827: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:20.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:19 smithi179 ceph-mon[63029]: pgmap v828: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:20.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:19 smithi084 ceph-mon[28081]: pgmap v828: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:20.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:19 smithi161 ceph-mon[62387]: pgmap v828: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:22.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:21 smithi179 ceph-mon[63029]: pgmap v829: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:22.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:21 smithi084 ceph-mon[28081]: pgmap v829: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:22.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:21 smithi161 ceph-mon[62387]: pgmap v829: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:24.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:23 smithi179 ceph-mon[63029]: pgmap v830: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:24.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:23 smithi084 ceph-mon[28081]: pgmap v830: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:24.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:23 smithi161 ceph-mon[62387]: pgmap v830: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:26.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:25 smithi179 ceph-mon[63029]: pgmap v831: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:26.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:25 smithi084 ceph-mon[28081]: pgmap v831: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:26.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:25 smithi161 ceph-mon[62387]: pgmap v831: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:28.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:27 smithi179 ceph-mon[63029]: pgmap v832: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:28.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:27 smithi084 ceph-mon[28081]: pgmap v832: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:28.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:27 smithi161 ceph-mon[62387]: pgmap v832: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:30.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:29 smithi179 ceph-mon[63029]: pgmap v833: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:30.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:29 smithi084 ceph-mon[28081]: pgmap v833: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:30.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:29 smithi161 ceph-mon[62387]: pgmap v833: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:32.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:31 smithi179 ceph-mon[63029]: pgmap v834: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:32.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:31 smithi084 ceph-mon[28081]: pgmap v834: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:32.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:31 smithi161 ceph-mon[62387]: pgmap v834: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:34.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:33 smithi179 ceph-mon[63029]: pgmap v835: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:34.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:33 smithi084 ceph-mon[28081]: pgmap v835: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:34.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:33 smithi161 ceph-mon[62387]: pgmap v835: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:36.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:35 smithi179 ceph-mon[63029]: pgmap v836: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:36.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:35 smithi084 ceph-mon[28081]: pgmap v836: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:36.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:35 smithi161 ceph-mon[62387]: pgmap v836: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:38.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:37 smithi179 ceph-mon[63029]: pgmap v837: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:38.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:37 smithi084 ceph-mon[28081]: pgmap v837: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:38.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:37 smithi161 ceph-mon[62387]: pgmap v837: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:40.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:39 smithi179 ceph-mon[63029]: pgmap v838: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:05:40.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:39 smithi084 ceph-mon[28081]: pgmap v838: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:05:40.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:39 smithi161 ceph-mon[62387]: pgmap v838: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:05:42.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:41 smithi179 ceph-mon[63029]: pgmap v839: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:05:42.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:41 smithi084 ceph-mon[28081]: pgmap v839: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:05:42.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:41 smithi161 ceph-mon[62387]: pgmap v839: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-28T13:05:44.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:43 smithi179 ceph-mon[63029]: pgmap v840: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:05:44.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:43 smithi084 ceph-mon[28081]: pgmap v840: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:05:44.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:43 smithi161 ceph-mon[62387]: pgmap v840: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-28T13:05:46.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:45 smithi179 ceph-mon[63029]: pgmap v841: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:05:46.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:45 smithi084 ceph-mon[28081]: pgmap v841: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:05:46.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:45 smithi161 ceph-mon[62387]: pgmap v841: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-28T13:05:48.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:47 smithi179 ceph-mon[63029]: pgmap v842: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:48.170 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:47 smithi084 ceph-mon[28081]: pgmap v842: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:48.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:47 smithi161 ceph-mon[62387]: pgmap v842: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:49.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:48 smithi179 ceph-mon[63029]: pgmap v843: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:49.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:48 smithi084 ceph-mon[28081]: pgmap v843: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:49.246 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:48 smithi161 ceph-mon[62387]: pgmap v843: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:51.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:51 smithi161 ceph-mon[62387]: pgmap v844: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:51.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:51 smithi179 ceph-mon[63029]: pgmap v844: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:51.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:51 smithi084 ceph-mon[28081]: pgmap v844: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:05:53.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:53 smithi161 ceph-mon[62387]: pgmap v845: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:53.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:53 smithi179 ceph-mon[63029]: pgmap v845: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:53.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:53 smithi084 ceph-mon[28081]: pgmap v845: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:05:55.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:55 smithi161 ceph-mon[62387]: pgmap v846: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:55.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:55 smithi179 ceph-mon[63029]: pgmap v846: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:55.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:55 smithi084 ceph-mon[28081]: pgmap v846: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:57.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:57 smithi161 ceph-mon[62387]: pgmap v847: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:57.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:57 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:05:57.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:57 smithi179 ceph-mon[63029]: pgmap v847: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:57.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:57 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:05:57.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:57 smithi084 ceph-mon[28081]: pgmap v847: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:05:57.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:57 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-28T13:05:59.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:59 smithi161 ceph-mon[62387]: pgmap v848: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:59.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:59 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:05:59.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:59 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:05:59.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:05:59 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:05:59.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:59 smithi179 ceph-mon[63029]: pgmap v848: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:59.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:59 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:05:59.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:59 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:05:59.669 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:05:59 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:05:59.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:59 smithi084 ceph-mon[28081]: pgmap v848: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:05:59.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-28T13:05:59.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-28T13:05:59.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:05:59 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' 2024-04-28T13:06:01.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:01 smithi179 ceph-mon[63029]: pgmap v849: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:01.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:01 smithi084 ceph-mon[28081]: pgmap v849: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:01.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:01 smithi161 ceph-mon[62387]: pgmap v849: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:03.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:03 smithi179 ceph-mon[63029]: pgmap v850: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:03.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:03 smithi084 ceph-mon[28081]: pgmap v850: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:03.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:03 smithi161 ceph-mon[62387]: pgmap v850: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:05.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:05 smithi179 ceph-mon[63029]: pgmap v851: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:05.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:05 smithi084 ceph-mon[28081]: pgmap v851: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:05.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:05 smithi161 ceph-mon[62387]: pgmap v851: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:07.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:07 smithi179 ceph-mon[63029]: pgmap v852: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:06:07.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:07 smithi084 ceph-mon[28081]: pgmap v852: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:06:07.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:07 smithi161 ceph-mon[62387]: pgmap v852: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:06:09.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:09 smithi179 ceph-mon[63029]: pgmap v853: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:09.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:09 smithi084 ceph-mon[28081]: pgmap v853: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:09.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:09 smithi161 ceph-mon[62387]: pgmap v853: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:11.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:11 smithi179 ceph-mon[63029]: pgmap v854: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:11.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:11 smithi084 ceph-mon[28081]: pgmap v854: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:11.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:11 smithi161 ceph-mon[62387]: pgmap v854: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:13.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:13 smithi179 ceph-mon[63029]: pgmap v855: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:13.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:13 smithi084 ceph-mon[28081]: pgmap v855: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:13.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:13 smithi161 ceph-mon[62387]: pgmap v855: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:15.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:15 smithi179 ceph-mon[63029]: pgmap v856: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:15.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:15 smithi084 ceph-mon[28081]: pgmap v856: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:15.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:15 smithi161 ceph-mon[62387]: pgmap v856: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:17.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:17 smithi179 ceph-mon[63029]: pgmap v857: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:06:17.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:17 smithi084 ceph-mon[28081]: pgmap v857: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:06:17.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:17 smithi161 ceph-mon[62387]: pgmap v857: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:06:19.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:19 smithi179 ceph-mon[63029]: pgmap v858: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:19.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:19 smithi084 ceph-mon[28081]: pgmap v858: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:19.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:19 smithi161 ceph-mon[62387]: pgmap v858: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:21.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:21 smithi179 ceph-mon[63029]: pgmap v859: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:21.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:21 smithi084 ceph-mon[28081]: pgmap v859: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:21.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:21 smithi161 ceph-mon[62387]: pgmap v859: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:23.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:23 smithi179 ceph-mon[63029]: pgmap v860: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:23.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:23 smithi084 ceph-mon[28081]: pgmap v860: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:23.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:23 smithi161 ceph-mon[62387]: pgmap v860: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:25.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:25 smithi179 ceph-mon[63029]: pgmap v861: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:25.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:25 smithi084 ceph-mon[28081]: pgmap v861: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:25.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:25 smithi161 ceph-mon[62387]: pgmap v861: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:27.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:27 smithi179 ceph-mon[63029]: pgmap v862: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:06:27.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:27 smithi084 ceph-mon[28081]: pgmap v862: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:06:27.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:27 smithi161 ceph-mon[62387]: pgmap v862: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:06:29.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:29 smithi179 ceph-mon[63029]: pgmap v863: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:29.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:29 smithi084 ceph-mon[28081]: pgmap v863: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:29.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:29 smithi161 ceph-mon[62387]: pgmap v863: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:31.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:31 smithi179 ceph-mon[63029]: pgmap v864: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:31.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:31 smithi084 ceph-mon[28081]: pgmap v864: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:31.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:31 smithi161 ceph-mon[62387]: pgmap v864: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:33.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:33 smithi179 ceph-mon[63029]: pgmap v865: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:33.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:33 smithi084 ceph-mon[28081]: pgmap v865: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:33.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:33 smithi161 ceph-mon[62387]: pgmap v865: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:35.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:35 smithi179 ceph-mon[63029]: pgmap v866: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:35.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:35 smithi084 ceph-mon[28081]: pgmap v866: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:35.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:35 smithi161 ceph-mon[62387]: pgmap v866: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:37.022 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.022+0000 7f08d7e00640 1 -- 172.21.15.84:0/2794133779 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f08bc0a3cb0 msgr2=0x7f08bc0a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900028015) for more than 900000000 us, fault. 2024-04-28T13:06:37.022 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.022+0000 7f08d7e00640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f08bc0a3cb0 0x7f08bc0a6170 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009fc0 tx=0x7f08c8005f90 comp rx=0 tx=0).stop 2024-04-28T13:06:37.022 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.022+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f08bc0a3cb0 msgr2=0x7f08bc0a6170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-28T13:06:37.022 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.022+0000 7f08d5600640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f08bc0a3cb0 0x7f08bc0a6170 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:37.022 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.022+0000 7f08d5600640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f08bc0af2e0 0x7f08bc0b16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:37.022 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.022+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 --> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f08bc0b1cb0 con 0x7f08bc0af2e0 2024-04-28T13:06:37.023 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.023+0000 7f08d7e00640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f08bc0af2e0 0x7f08bc0b16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:37.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.024+0000 7f08d7e00640 1 --2- 172.21.15.84:0/2794133779 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f08bc0af2e0 0x7f08bc0b16d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f08c8013010 tx=0x7f08c8018000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:37.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.136+0000 7f08cf400640 1 -- 172.21.15.84:0/2794133779 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f08b0006380 con 0x7f08d810ed90 2024-04-28T13:06:37.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.137+0000 7f08d5600640 1 -- 172.21.15.84:0/2794133779 <== mon.0 v2:172.21.15.84:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f08b0006380 con 0x7f08d810ed90 2024-04-28T13:06:37.228 INFO:teuthology.orchestra.run.smithi084.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-04-28T13:06:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-28T13:06:37.246 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-28T13:06:37.486 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=dc88b918d941469280c7aa2c8a1c6e81 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi084 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-28T13:06:37.488 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-28T13:06:37.498 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-28T13:06:37.498 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T13:06:37.498 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T13:06:37.559 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-04-28T13:06:37.559 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-28T13:06:37.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:06:37.808 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.808+0000 7f4493200640 1 -- 172.21.15.84:0/3760920901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f448c103210 msgr2=0x7f448c105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:37.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.808+0000 7f4493200640 1 --2- 172.21.15.84:0/3760920901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f448c103210 0x7f448c105600 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4480009900 tx=0x7f448002a410 comp rx=0 tx=0).stop 2024-04-28T13:06:37.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.809+0000 7f4493200640 1 -- 172.21.15.84:0/3760920901 shutdown_connections 2024-04-28T13:06:37.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.809+0000 7f4493200640 1 --2- 172.21.15.84:0/3760920901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f448c108470 0x7f448c10a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:37.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.809+0000 7f4493200640 1 --2- 172.21.15.84:0/3760920901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f448c105b40 0x7f448c107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:37.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.809+0000 7f4493200640 1 --2- 172.21.15.84:0/3760920901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f448c103210 0x7f448c105600 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:37.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.809+0000 7f4493200640 1 -- 172.21.15.84:0/3760920901 >> 172.21.15.84:0/3760920901 conn(0x7f448c0fcc40 msgr2=0x7f448c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:37.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.809+0000 7f4493200640 1 -- 172.21.15.84:0/3760920901 shutdown_connections 2024-04-28T13:06:37.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.809+0000 7f4493200640 1 -- 172.21.15.84:0/3760920901 wait complete. 2024-04-28T13:06:37.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.810+0000 7f4493200640 1 Processor -- start 2024-04-28T13:06:37.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.810+0000 7f4493200640 1 -- start start 2024-04-28T13:06:37.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.810+0000 7f4493200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f448c103210 0x7f448c19abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:37.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.810+0000 7f4493200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f448c105b40 0x7f448c19b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:37.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.810+0000 7f4493200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f448c108470 0x7f448c1a2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:37.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.810+0000 7f4493200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f448c10e340 con 0x7f448c108470 2024-04-28T13:06:37.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.810+0000 7f4493200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f448c10e1c0 con 0x7f448c103210 2024-04-28T13:06:37.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.810+0000 7f4493200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f448c10e4c0 con 0x7f448c105b40 2024-04-28T13:06:37.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.810+0000 7f4492800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f448c108470 0x7f448c1a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:37.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.810+0000 7f4491e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f448c103210 0x7f448c19abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:37.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.810+0000 7f4492800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f448c108470 0x7f448c1a2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35016/0 (socket says 172.21.15.84:35016) 2024-04-28T13:06:37.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.810+0000 7f4492800640 1 -- 172.21.15.84:0/1752475150 learned_addr learned my addr 172.21.15.84:0/1752475150 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:37.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.811+0000 7f4491400640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f448c105b40 0x7f448c19b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:37.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.811+0000 7f4492800640 1 -- 172.21.15.84:0/1752475150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f448c105b40 msgr2=0x7f448c19b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:37.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.811+0000 7f4492800640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f448c105b40 0x7f448c19b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:37.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.811+0000 7f4492800640 1 -- 172.21.15.84:0/1752475150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f448c103210 msgr2=0x7f448c19abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:37.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.811+0000 7f4492800640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f448c103210 0x7f448c19abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:37.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.811+0000 7f4492800640 1 -- 172.21.15.84:0/1752475150 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f448c1a27f0 con 0x7f448c108470 2024-04-28T13:06:37.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.811+0000 7f4492800640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f448c108470 0x7f448c1a2180 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f448800eb10 tx=0x7f448800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:37.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.811+0000 7f4486a00640 1 -- 172.21.15.84:0/1752475150 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488004b20 con 0x7f448c108470 2024-04-28T13:06:37.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.811+0000 7f4486a00640 1 -- 172.21.15.84:0/1752475150 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4488004cc0 con 0x7f448c108470 2024-04-28T13:06:37.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.811+0000 7f4493200640 1 -- 172.21.15.84:0/1752475150 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f448c1a2ae0 con 0x7f448c108470 2024-04-28T13:06:37.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.811+0000 7f4486a00640 1 -- 172.21.15.84:0/1752475150 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488010670 con 0x7f448c108470 2024-04-28T13:06:37.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.812+0000 7f4493200640 1 -- 172.21.15.84:0/1752475150 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f448c1a2fe0 con 0x7f448c108470 2024-04-28T13:06:37.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.813+0000 7f4493200640 1 -- 172.21.15.84:0/1752475150 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f448c109fc0 con 0x7f448c108470 2024-04-28T13:06:37.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.815+0000 7f4486a00640 1 -- 172.21.15.84:0/1752475150 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4488005020 con 0x7f448c108470 2024-04-28T13:06:37.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.817+0000 7f4486a00640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f44740a3c40 0x7f44740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:37.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.817+0000 7f4486a00640 1 -- 172.21.15.84:0/1752475150 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f44880d0f20 con 0x7f448c108470 2024-04-28T13:06:37.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.817+0000 7f4491e00640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f44740a3c40 0x7f44740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:37.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.818+0000 7f4491e00640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f44740a3c40 0x7f44740a6100 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4480031500 tx=0x7f4480005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:37.821 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:37.822+0000 7f4486a00640 1 -- 172.21.15.84:0/1752475150 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f448809d3c0 con 0x7f448c108470 2024-04-28T13:06:37.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:37 smithi179 ceph-mon[63029]: pgmap v867: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:06:37.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:37 smithi084 ceph-mon[28081]: pgmap v867: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:06:37.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:37 smithi161 ceph-mon[62387]: pgmap v867: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-28T13:06:38.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.026+0000 7f4493200640 1 -- 172.21.15.84:0/1752475150 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f448c02dcf0 con 0x7f448c108470 2024-04-28T13:06:38.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.030+0000 7f4486a00640 1 -- 172.21.15.84:0/1752475150 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24634 (secure 0 0 0) 0x7f448809ccc0 con 0x7f448c108470 2024-04-28T13:06:38.031 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T13:06:38.031 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":94,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:50:30.848698+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.8","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:06:38.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.033+0000 7f4493200640 1 -- 172.21.15.84:0/1752475150 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f44740a3c40 msgr2=0x7f44740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:38.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.033+0000 7f4493200640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f44740a3c40 0x7f44740a6100 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4480031500 tx=0x7f4480005a20 comp rx=0 tx=0).stop 2024-04-28T13:06:38.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.033+0000 7f4493200640 1 -- 172.21.15.84:0/1752475150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f448c108470 msgr2=0x7f448c1a2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:38.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.033+0000 7f4493200640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f448c108470 0x7f448c1a2180 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f448800eb10 tx=0x7f448800c2d0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.033+0000 7f4493200640 1 -- 172.21.15.84:0/1752475150 shutdown_connections 2024-04-28T13:06:38.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.033+0000 7f4493200640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f44740a3c40 0x7f44740a6100 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.033+0000 7f4493200640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f448c108470 0x7f448c1a2180 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.033+0000 7f4493200640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f448c105b40 0x7f448c19b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.033+0000 7f4493200640 1 --2- 172.21.15.84:0/1752475150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f448c103210 0x7f448c19abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.033+0000 7f4493200640 1 -- 172.21.15.84:0/1752475150 >> 172.21.15.84:0/1752475150 conn(0x7f448c0fcc40 msgr2=0x7f448c1029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:38.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.034+0000 7f4493200640 1 -- 172.21.15.84:0/1752475150 shutdown_connections 2024-04-28T13:06:38.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.034+0000 7f4493200640 1 -- 172.21.15.84:0/1752475150 wait complete. 2024-04-28T13:06:38.053 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-28T12:43:15.608080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-28T12:48:32.037809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-28T12:48:33.240166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-28T12:48:37.567238+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-28T13:06:38.053 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-28T13:06:38.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.258+0000 7f51fa000640 1 -- 172.21.15.84:0/878786615 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51f4106620 msgr2=0x7f51f410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:38.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.258+0000 7f51fa000640 1 --2- 172.21.15.84:0/878786615 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51f4106620 0x7f51f410ceb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009ca0 tx=0x7f51ec02a8c0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.258+0000 7f51fa000640 1 -- 172.21.15.84:0/878786615 shutdown_connections 2024-04-28T13:06:38.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.258+0000 7f51fa000640 1 --2- 172.21.15.84:0/878786615 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51f4106620 0x7f51f410ceb0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.258+0000 7f51fa000640 1 --2- 172.21.15.84:0/878786615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51f4105c60 0x7f51f41060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.258+0000 7f51fa000640 1 --2- 172.21.15.84:0/878786615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4104a70 0x7f51f4104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.258+0000 7f51fa000640 1 -- 172.21.15.84:0/878786615 >> 172.21.15.84:0/878786615 conn(0x7f51f40fffe0 msgr2=0x7f51f4102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:38.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.258+0000 7f51fa000640 1 -- 172.21.15.84:0/878786615 shutdown_connections 2024-04-28T13:06:38.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.258+0000 7f51fa000640 1 -- 172.21.15.84:0/878786615 wait complete. 2024-04-28T13:06:38.261 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.259+0000 7f51fa000640 1 Processor -- start 2024-04-28T13:06:38.261 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.260+0000 7f51fa000640 1 -- start start 2024-04-28T13:06:38.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.260+0000 7f51fa000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4104a70 0x7f51f419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:38.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.260+0000 7f51fa000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51f4105c60 0x7f51f419b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:38.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.260+0000 7f51fa000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51f4106620 0x7f51f41a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:38.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.260+0000 7f51fa000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f51f41105d0 con 0x7f51f4105c60 2024-04-28T13:06:38.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.260+0000 7f51fa000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f51f4110450 con 0x7f51f4104a70 2024-04-28T13:06:38.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.260+0000 7f51fa000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f51f4110750 con 0x7f51f4106620 2024-04-28T13:06:38.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.260+0000 7f51f3e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51f4105c60 0x7f51f419b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:38.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.260+0000 7f51f3e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51f4105c60 0x7f51f419b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35036/0 (socket says 172.21.15.84:35036) 2024-04-28T13:06:38.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.261+0000 7f51f3e00640 1 -- 172.21.15.84:0/1321935770 learned_addr learned my addr 172.21.15.84:0/1321935770 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:38.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.261+0000 7f51f8c00640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4104a70 0x7f51f419abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:38.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.261+0000 7f51f9600640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51f4106620 0x7f51f41a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:38.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.261+0000 7f51f3e00640 1 -- 172.21.15.84:0/1321935770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51f4106620 msgr2=0x7f51f41a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:38.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.261+0000 7f51f3e00640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51f4106620 0x7f51f41a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.261+0000 7f51f3e00640 1 -- 172.21.15.84:0/1321935770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4104a70 msgr2=0x7f51f419abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:38.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.261+0000 7f51f3e00640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4104a70 0x7f51f419abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.261+0000 7f51f3e00640 1 -- 172.21.15.84:0/1321935770 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f41a2810 con 0x7f51f4105c60 2024-04-28T13:06:38.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.261+0000 7f51f3e00640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51f4105c60 0x7f51f419b130 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f51e400e9f0 tx=0x7f51e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:38.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.261+0000 7f51f1600640 1 -- 172.21.15.84:0/1321935770 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e400cd80 con 0x7f51f4105c60 2024-04-28T13:06:38.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.261+0000 7f51f1600640 1 -- 172.21.15.84:0/1321935770 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51e40108e0 con 0x7f51f4105c60 2024-04-28T13:06:38.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.261+0000 7f51f1600640 1 -- 172.21.15.84:0/1321935770 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e4018980 con 0x7f51f4105c60 2024-04-28T13:06:38.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.262+0000 7f51fa000640 1 -- 172.21.15.84:0/1321935770 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f41a2b00 con 0x7f51f4105c60 2024-04-28T13:06:38.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.262+0000 7f51fa000640 1 -- 172.21.15.84:0/1321935770 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f41a30a0 con 0x7f51f4105c60 2024-04-28T13:06:38.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.263+0000 7f51fa000640 1 -- 172.21.15.84:0/1321935770 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f410bd90 con 0x7f51f4105c60 2024-04-28T13:06:38.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.265+0000 7f51f1600640 1 -- 172.21.15.84:0/1321935770 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f51e4010430 con 0x7f51f4105c60 2024-04-28T13:06:38.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.266+0000 7f51f1600640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f51dc0a3cc0 0x7f51dc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:38.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.267+0000 7f51f1600640 1 -- 172.21.15.84:0/1321935770 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f51e40d0c80 con 0x7f51f4105c60 2024-04-28T13:06:38.266 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.267+0000 7f51f8c00640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f51dc0a3cc0 0x7f51dc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:38.267 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.268+0000 7f51f8c00640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f51dc0a3cc0 0x7f51dc0a6180 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f51f4105ac0 tx=0x7f51e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:38.271 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.271+0000 7f51f1600640 1 -- 172.21.15.84:0/1321935770 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f51e409d120 con 0x7f51f4105c60 2024-04-28T13:06:38.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.475+0000 7f51fa000640 1 -- 172.21.15.84:0/1321935770 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f51f402dd10 con 0x7f51f4105c60 2024-04-28T13:06:38.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.476+0000 7f51f1600640 1 -- 172.21.15.84:0/1321935770 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f51e409ca20 con 0x7f51f4105c60 2024-04-28T13:06:38.475 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-04-28T13:06:38.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.479+0000 7f51fa000640 1 -- 172.21.15.84:0/1321935770 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f51dc0a3cc0 msgr2=0x7f51dc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:38.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.479+0000 7f51fa000640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f51dc0a3cc0 0x7f51dc0a6180 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f51f4105ac0 tx=0x7f51e8005f90 comp rx=0 tx=0).stop 2024-04-28T13:06:38.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.479+0000 7f51fa000640 1 -- 172.21.15.84:0/1321935770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51f4105c60 msgr2=0x7f51f419b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:38.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.479+0000 7f51fa000640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51f4105c60 0x7f51f419b130 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f51e400e9f0 tx=0x7f51e400eeb0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.479+0000 7f51fa000640 1 -- 172.21.15.84:0/1321935770 shutdown_connections 2024-04-28T13:06:38.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.479+0000 7f51fa000640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f51dc0a3cc0 0x7f51dc0a6180 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.479+0000 7f51fa000640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f51f4106620 0x7f51f41a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.479+0000 7f51fa000640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51f4105c60 0x7f51f419b130 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.479+0000 7f51fa000640 1 --2- 172.21.15.84:0/1321935770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51f4104a70 0x7f51f419abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.479+0000 7f51fa000640 1 -- 172.21.15.84:0/1321935770 >> 172.21.15.84:0/1321935770 conn(0x7f51f40fffe0 msgr2=0x7f51f4108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:38.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.479+0000 7f51fa000640 1 -- 172.21.15.84:0/1321935770 shutdown_connections 2024-04-28T13:06:38.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.479+0000 7f51fa000640 1 -- 172.21.15.84:0/1321935770 wait complete. 2024-04-28T13:06:38.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-28T13:06:38.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.706+0000 7fd5cee00640 1 -- 172.21.15.84:0/3209189837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c80772f0 msgr2=0x7fd5c8077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:38.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.706+0000 7fd5cee00640 1 --2- 172.21.15.84:0/3209189837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c80772f0 0x7fd5c8077710 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009900 tx=0x7fd5b802a410 comp rx=0 tx=0).stop 2024-04-28T13:06:38.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.706+0000 7fd5cee00640 1 -- 172.21.15.84:0/3209189837 shutdown_connections 2024-04-28T13:06:38.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.706+0000 7fd5cee00640 1 --2- 172.21.15.84:0/3209189837 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c8078ed0 0x7fd5c8079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.706+0000 7fd5cee00640 1 --2- 172.21.15.84:0/3209189837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5c8078530 0x7fd5c8078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.706+0000 7fd5cee00640 1 --2- 172.21.15.84:0/3209189837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c80772f0 0x7fd5c8077710 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.706+0000 7fd5cee00640 1 -- 172.21.15.84:0/3209189837 >> 172.21.15.84:0/3209189837 conn(0x7fd5c81000e0 msgr2=0x7fd5c8102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:38.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.706+0000 7fd5cee00640 1 -- 172.21.15.84:0/3209189837 shutdown_connections 2024-04-28T13:06:38.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.706+0000 7fd5cee00640 1 -- 172.21.15.84:0/3209189837 wait complete. 2024-04-28T13:06:38.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.707+0000 7fd5cee00640 1 Processor -- start 2024-04-28T13:06:38.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.707+0000 7fd5cee00640 1 -- start start 2024-04-28T13:06:38.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cee00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5c80772f0 0x7fd5c819f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:38.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cee00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c8078530 0x7fd5c819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:38.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cee00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c8078ed0 0x7fd5c81a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:38.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cee00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c8114b10 con 0x7fd5c8078ed0 2024-04-28T13:06:38.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cee00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c8114990 con 0x7fd5c80772f0 2024-04-28T13:06:38.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cee00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c8114c90 con 0x7fd5c8078530 2024-04-28T13:06:38.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cce00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c8078ed0 0x7fd5c81a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:38.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5c7e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5c80772f0 0x7fd5c819f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:38.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cce00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c8078ed0 0x7fd5c81a6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35060/0 (socket says 172.21.15.84:35060) 2024-04-28T13:06:38.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cce00640 1 -- 172.21.15.84:0/1463995460 learned_addr learned my addr 172.21.15.84:0/1463995460 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:38.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5bf400640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c8078530 0x7fd5c819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:38.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cce00640 1 -- 172.21.15.84:0/1463995460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c8078530 msgr2=0x7fd5c819f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:38.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cce00640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c8078530 0x7fd5c819f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cce00640 1 -- 172.21.15.84:0/1463995460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5c80772f0 msgr2=0x7fd5c819f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:38.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cce00640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5c80772f0 0x7fd5c819f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.708+0000 7fd5cce00640 1 -- 172.21.15.84:0/1463995460 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c81a6de0 con 0x7fd5c8078ed0 2024-04-28T13:06:38.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.709+0000 7fd5cce00640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c8078ed0 0x7fd5c81a6770 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000eb10 tx=0x7fd5c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:38.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.709+0000 7fd5c5600640 1 -- 172.21.15.84:0/1463995460 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0004b20 con 0x7fd5c8078ed0 2024-04-28T13:06:38.712 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.709+0000 7fd5c5600640 1 -- 172.21.15.84:0/1463995460 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5c0004cc0 con 0x7fd5c8078ed0 2024-04-28T13:06:38.712 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.709+0000 7fd5c5600640 1 -- 172.21.15.84:0/1463995460 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0010670 con 0x7fd5c8078ed0 2024-04-28T13:06:38.712 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.709+0000 7fd5cee00640 1 -- 172.21.15.84:0/1463995460 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c81a70d0 con 0x7fd5c8078ed0 2024-04-28T13:06:38.712 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.709+0000 7fd5cee00640 1 -- 172.21.15.84:0/1463995460 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c81a7670 con 0x7fd5c8078ed0 2024-04-28T13:06:38.712 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.710+0000 7fd5cee00640 1 -- 172.21.15.84:0/1463995460 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5c8111700 con 0x7fd5c8078ed0 2024-04-28T13:06:38.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.712+0000 7fd5c5600640 1 -- 172.21.15.84:0/1463995460 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd5c0005020 con 0x7fd5c8078ed0 2024-04-28T13:06:38.714 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.715+0000 7fd5c5600640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5ac0a3cc0 0x7fd5ac0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:38.714 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.715+0000 7fd5c5600640 1 -- 172.21.15.84:0/1463995460 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fd5c00d0e10 con 0x7fd5c8078ed0 2024-04-28T13:06:38.715 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.715+0000 7fd5c7e00640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5ac0a3cc0 0x7fd5ac0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:38.715 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.716+0000 7fd5c7e00640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5ac0a3cc0 0x7fd5ac0a6180 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8031500 tx=0x7fd5b8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:38.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.719+0000 7fd5c5600640 1 -- 172.21.15.84:0/1463995460 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd5c00d4030 con 0x7fd5c8078ed0 2024-04-28T13:06:38.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:38 smithi179 ceph-mon[63029]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:06:38.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:38 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1752475150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:06:38.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:38 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1321935770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:06:38.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:38 smithi084 ceph-mon[28081]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:06:38.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:38 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1752475150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:06:38.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:38 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1321935770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:06:38.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.924+0000 7fd5cee00640 1 -- 172.21.15.84:0/1463995460 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd5c8063530 con 0x7fd5c8078ed0 2024-04-28T13:06:38.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.925+0000 7fd5c5600640 1 -- 172.21.15.84:0/1463995460 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fd5c009d2b0 con 0x7fd5c8078ed0 2024-04-28T13:06:38.925 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T13:06:38.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.928+0000 7fd5cee00640 1 -- 172.21.15.84:0/1463995460 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5ac0a3cc0 msgr2=0x7fd5ac0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:38.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.928+0000 7fd5cee00640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5ac0a3cc0 0x7fd5ac0a6180 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8031500 tx=0x7fd5b8005a20 comp rx=0 tx=0).stop 2024-04-28T13:06:38.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.928+0000 7fd5cee00640 1 -- 172.21.15.84:0/1463995460 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c8078ed0 msgr2=0x7fd5c81a6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:38.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.928+0000 7fd5cee00640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c8078ed0 0x7fd5c81a6770 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000eb10 tx=0x7fd5c000c2d0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.928+0000 7fd5cee00640 1 -- 172.21.15.84:0/1463995460 shutdown_connections 2024-04-28T13:06:38.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.928+0000 7fd5cee00640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fd5ac0a3cc0 0x7fd5ac0a6180 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.928+0000 7fd5cee00640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5c8078ed0 0x7fd5c81a6770 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.928+0000 7fd5cee00640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5c8078530 0x7fd5c819f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.928+0000 7fd5cee00640 1 --2- 172.21.15.84:0/1463995460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5c80772f0 0x7fd5c819f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:38.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.928+0000 7fd5cee00640 1 -- 172.21.15.84:0/1463995460 >> 172.21.15.84:0/1463995460 conn(0x7fd5c81000e0 msgr2=0x7fd5c8101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:38.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.928+0000 7fd5cee00640 1 -- 172.21.15.84:0/1463995460 shutdown_connections 2024-04-28T13:06:38.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:38.928+0000 7fd5cee00640 1 -- 172.21.15.84:0/1463995460 wait complete. 2024-04-28T13:06:38.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-28T13:06:38.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:38 smithi161 ceph-mon[62387]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-28T13:06:38.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:38 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1752475150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:06:38.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:38 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1321935770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-28T13:06:39.149 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.149+0000 7f8019800640 1 -- 172.21.15.84:0/2451934349 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8014101ff0 msgr2=0x7f8014102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:39.149 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.149+0000 7f8019800640 1 --2- 172.21.15.84:0/2451934349 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8014101ff0 0x7f8014102470 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc009900 tx=0x7f7ffc02a3f0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.149 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.149+0000 7f8019800640 1 -- 172.21.15.84:0/2451934349 shutdown_connections 2024-04-28T13:06:39.149 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.149+0000 7f8019800640 1 --2- 172.21.15.84:0/2451934349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80141029b0 0x7f8014109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.149 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.149+0000 7f8019800640 1 --2- 172.21.15.84:0/2451934349 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8014101ff0 0x7f8014102470 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.149 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.149+0000 7f8019800640 1 --2- 172.21.15.84:0/2451934349 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8014100e00 0x7f8014101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.150 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.149+0000 7f8019800640 1 -- 172.21.15.84:0/2451934349 >> 172.21.15.84:0/2451934349 conn(0x7f80140fc370 msgr2=0x7f80140fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:39.150 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.149+0000 7f8019800640 1 -- 172.21.15.84:0/2451934349 shutdown_connections 2024-04-28T13:06:39.150 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.149+0000 7f8019800640 1 -- 172.21.15.84:0/2451934349 wait complete. 2024-04-28T13:06:39.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.150+0000 7f8019800640 1 Processor -- start 2024-04-28T13:06:39.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.150+0000 7f8019800640 1 -- start start 2024-04-28T13:06:39.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8019800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8014100e00 0x7f8014198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:39.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8019800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8014101ff0 0x7f8014198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:39.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8019800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80141029b0 0x7f801419ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:39.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8019800640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f801410c0f0 con 0x7f80141029b0 2024-04-28T13:06:39.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8019800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f801410bf70 con 0x7f8014101ff0 2024-04-28T13:06:39.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8019800640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f801410c270 con 0x7f8014100e00 2024-04-28T13:06:39.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8013e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8014100e00 0x7f8014198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:39.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f800be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8014101ff0 0x7f8014198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:39.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8018e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80141029b0 0x7f801419ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:39.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8013e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8014100e00 0x7f8014198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:43950/0 (socket says 172.21.15.84:43950) 2024-04-28T13:06:39.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8013e00640 1 -- 172.21.15.84:0/647824172 learned_addr learned my addr 172.21.15.84:0/647824172 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:39.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8013e00640 1 -- 172.21.15.84:0/647824172 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8014101ff0 msgr2=0x7f8014198fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:39.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8013e00640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8014101ff0 0x7f8014198fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8013e00640 1 -- 172.21.15.84:0/647824172 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80141029b0 msgr2=0x7f801419ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:39.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8013e00640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80141029b0 0x7f801419ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8013e00640 1 -- 172.21.15.84:0/647824172 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80141a0640 con 0x7f8014100e00 2024-04-28T13:06:39.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f8018e00640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80141029b0 0x7f801419ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:06:39.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.151+0000 7f800be00640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8014101ff0 0x7f8014198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:06:39.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.152+0000 7f8013e00640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8014100e00 0x7f8014198a70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f800400e9c0 tx=0x7f800400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:39.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.152+0000 7f8011600640 1 -- 172.21.15.84:0/647824172 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800400cd80 con 0x7f8014100e00 2024-04-28T13:06:39.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.152+0000 7f8019800640 1 -- 172.21.15.84:0/647824172 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80141a0930 con 0x7f8014100e00 2024-04-28T13:06:39.155 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.152+0000 7f8011600640 1 -- 172.21.15.84:0/647824172 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80040035f0 con 0x7f8014100e00 2024-04-28T13:06:39.155 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.152+0000 7f8011600640 1 -- 172.21.15.84:0/647824172 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8004010690 con 0x7f8014100e00 2024-04-28T13:06:39.155 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.152+0000 7f8019800640 1 -- 172.21.15.84:0/647824172 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80141a0e00 con 0x7f8014100e00 2024-04-28T13:06:39.155 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.153+0000 7f8019800640 1 -- 172.21.15.84:0/647824172 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80141a1030 con 0x7f8014100e00 2024-04-28T13:06:39.155 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.155+0000 7f8011600640 1 -- 172.21.15.84:0/647824172 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8004004000 con 0x7f8014100e00 2024-04-28T13:06:39.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.157+0000 7f8011600640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ff80a3c40 0x7f7ff80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:39.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.157+0000 7f8011600640 1 -- 172.21.15.84:0/647824172 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f80040d0e50 con 0x7f8014100e00 2024-04-28T13:06:39.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.157+0000 7f800be00640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ff80a3c40 0x7f7ff80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:39.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.158+0000 7f800be00640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ff80a3c40 0x7f7ff80a6100 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc031500 tx=0x7f7ffc005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:39.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.161+0000 7f8011600640 1 -- 172.21.15.84:0/647824172 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f800409d2f0 con 0x7f8014100e00 2024-04-28T13:06:39.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.362+0000 7f8019800640 1 -- 172.21.15.84:0/647824172 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f80141a12a0 con 0x7f8014100e00 2024-04-28T13:06:39.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.363+0000 7f8011600640 1 -- 172.21.15.84:0/647824172 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f800409cbf0 con 0x7f8014100e00 2024-04-28T13:06:39.363 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 64 2024-04-28T13:06:39.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.366+0000 7f8019800640 1 -- 172.21.15.84:0/647824172 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ff80a3c40 msgr2=0x7f7ff80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:39.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.366+0000 7f8019800640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ff80a3c40 0x7f7ff80a6100 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc031500 tx=0x7f7ffc005a20 comp rx=0 tx=0).stop 2024-04-28T13:06:39.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.366+0000 7f8019800640 1 -- 172.21.15.84:0/647824172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8014100e00 msgr2=0x7f8014198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:39.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.366+0000 7f8019800640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8014100e00 0x7f8014198a70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f800400e9c0 tx=0x7f800400ee80 comp rx=0 tx=0).stop 2024-04-28T13:06:39.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.366+0000 7f8019800640 1 -- 172.21.15.84:0/647824172 shutdown_connections 2024-04-28T13:06:39.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.366+0000 7f8019800640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f7ff80a3c40 0x7f7ff80a6100 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.366+0000 7f8019800640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f80141029b0 0x7f801419ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.366+0000 7f8019800640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8014101ff0 0x7f8014198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.366+0000 7f8019800640 1 --2- 172.21.15.84:0/647824172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8014100e00 0x7f8014198a70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.366+0000 7f8019800640 1 -- 172.21.15.84:0/647824172 >> 172.21.15.84:0/647824172 conn(0x7f80140fc370 msgr2=0x7f8014107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:39.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.366+0000 7f8019800640 1 -- 172.21.15.84:0/647824172 shutdown_connections 2024-04-28T13:06:39.367 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.366+0000 7f8019800640 1 -- 172.21.15.84:0/647824172 wait complete. 2024-04-28T13:06:39.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-28T13:06:39.590 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.590+0000 7f6567200640 1 -- 172.21.15.84:0/3508067632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6568101c70 msgr2=0x7f65681020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:39.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.590+0000 7f6567200640 1 --2- 172.21.15.84:0/3508067632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6568101c70 0x7f65681020f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6550009900 tx=0x7f655002a410 comp rx=0 tx=0).stop 2024-04-28T13:06:39.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.590+0000 7f6567200640 1 -- 172.21.15.84:0/3508067632 shutdown_connections 2024-04-28T13:06:39.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.590+0000 7f6567200640 1 --2- 172.21.15.84:0/3508067632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6568102630 0x7f6568108ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.590+0000 7f6567200640 1 --2- 172.21.15.84:0/3508067632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6568101c70 0x7f65681020f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.590+0000 7f6567200640 1 --2- 172.21.15.84:0/3508067632 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6568100a80 0x7f6568100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.590+0000 7f6567200640 1 -- 172.21.15.84:0/3508067632 >> 172.21.15.84:0/3508067632 conn(0x7f65680fbff0 msgr2=0x7f65680fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:39.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.590+0000 7f6567200640 1 -- 172.21.15.84:0/3508067632 shutdown_connections 2024-04-28T13:06:39.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.590+0000 7f6567200640 1 -- 172.21.15.84:0/3508067632 wait complete. 2024-04-28T13:06:39.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.591+0000 7f6567200640 1 Processor -- start 2024-04-28T13:06:39.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f6567200640 1 -- start start 2024-04-28T13:06:39.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f6567200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6568100a80 0x7f6568198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:39.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f6567200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6568101c70 0x7f6568198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:39.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f6567200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6568102630 0x7f65681a0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:39.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f6567200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f656810e380 con 0x7f6568101c70 2024-04-28T13:06:39.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f6567200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f656810e200 con 0x7f6568100a80 2024-04-28T13:06:39.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f6567200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f656810e500 con 0x7f6568102630 2024-04-28T13:06:39.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f655d400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6568101c70 0x7f6568198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:39.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f6565e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6568100a80 0x7f6568198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:39.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f6566800640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6568102630 0x7f65681a0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:39.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f655d400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6568101c70 0x7f6568198fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:46262/0 (socket says 172.21.15.84:46262) 2024-04-28T13:06:39.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f655d400640 1 -- 172.21.15.84:0/2870983712 learned_addr learned my addr 172.21.15.84:0/2870983712 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:39.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.592+0000 7f6566800640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6568102630 0x7f65681a0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.84:43966/0 (socket says 172.21.15.84:43966) 2024-04-28T13:06:39.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.593+0000 7f6566800640 1 -- 172.21.15.84:0/2870983712 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6568100a80 msgr2=0x7f6568198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:39.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.593+0000 7f6566800640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6568100a80 0x7f6568198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.593+0000 7f6566800640 1 -- 172.21.15.84:0/2870983712 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6568101c70 msgr2=0x7f6568198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:39.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.593+0000 7f6566800640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6568101c70 0x7f6568198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.593+0000 7f6566800640 1 -- 172.21.15.84:0/2870983712 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65681a06c0 con 0x7f6568102630 2024-04-28T13:06:39.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.593+0000 7f6565e00640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6568100a80 0x7f6568198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:06:39.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.593+0000 7f655d400640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6568101c70 0x7f6568198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:06:39.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.593+0000 7f6566800640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6568102630 0x7f65681a0050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6560009fd0 tx=0x7f656000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:39.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.593+0000 7f655f400640 1 -- 172.21.15.84:0/2870983712 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656000ce20 con 0x7f6568102630 2024-04-28T13:06:39.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.593+0000 7f6567200640 1 -- 172.21.15.84:0/2870983712 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f656810ae00 con 0x7f6568102630 2024-04-28T13:06:39.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.594+0000 7f655f400640 1 -- 172.21.15.84:0/2870983712 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6560005020 con 0x7f6568102630 2024-04-28T13:06:39.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.594+0000 7f655f400640 1 -- 172.21.15.84:0/2870983712 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560010690 con 0x7f6568102630 2024-04-28T13:06:39.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.594+0000 7f6567200640 1 -- 172.21.15.84:0/2870983712 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f656810b320 con 0x7f6568102630 2024-04-28T13:06:39.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.595+0000 7f6567200640 1 -- 172.21.15.84:0/2870983712 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6568107e90 con 0x7f6568102630 2024-04-28T13:06:39.598 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.597+0000 7f655f400640 1 -- 172.21.15.84:0/2870983712 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f65600054d0 con 0x7f6568102630 2024-04-28T13:06:39.598 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.598+0000 7f655f400640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f654c0a3c40 0x7f654c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:39.598 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.598+0000 7f655f400640 1 -- 172.21.15.84:0/2870983712 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f65600d0ff0 con 0x7f6568102630 2024-04-28T13:06:39.598 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.598+0000 7f6565e00640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f654c0a3c40 0x7f654c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.599+0000 7f6565e00640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f654c0a3c40 0x7f654c0a6100 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6568101ad0 tx=0x7f6558005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.600+0000 7f655f400640 1 -- 172.21.15.84:0/2870983712 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f656009d4b0 con 0x7f6568102630 2024-04-28T13:06:39.807 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.807+0000 7f6567200640 1 -- 172.21.15.84:0/2870983712 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f656802dad0 con 0x7f6568102630 2024-04-28T13:06:39.808 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.808+0000 7f655f400640 1 -- 172.21.15.84:0/2870983712 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f656009cdb0 con 0x7f6568102630 2024-04-28T13:06:39.808 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 32 2024-04-28T13:06:39.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.811+0000 7f6567200640 1 -- 172.21.15.84:0/2870983712 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f654c0a3c40 msgr2=0x7f654c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:39.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.811+0000 7f6567200640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f654c0a3c40 0x7f654c0a6100 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6568101ad0 tx=0x7f6558005f90 comp rx=0 tx=0).stop 2024-04-28T13:06:39.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.811+0000 7f6567200640 1 -- 172.21.15.84:0/2870983712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6568102630 msgr2=0x7f65681a0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:39.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.811+0000 7f6567200640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6568102630 0x7f65681a0050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6560009fd0 tx=0x7f656000ef10 comp rx=0 tx=0).stop 2024-04-28T13:06:39.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.811+0000 7f6567200640 1 -- 172.21.15.84:0/2870983712 shutdown_connections 2024-04-28T13:06:39.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.811+0000 7f6567200640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f654c0a3c40 0x7f654c0a6100 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.811+0000 7f6567200640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6568102630 0x7f65681a0050 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.811+0000 7f6567200640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6568101c70 0x7f6568198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.811+0000 7f6567200640 1 --2- 172.21.15.84:0/2870983712 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6568100a80 0x7f6568198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:39.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.811+0000 7f6567200640 1 -- 172.21.15.84:0/2870983712 >> 172.21.15.84:0/2870983712 conn(0x7f65680fbff0 msgr2=0x7f6568105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:39.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.811+0000 7f6567200640 1 -- 172.21.15.84:0/2870983712 shutdown_connections 2024-04-28T13:06:39.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:39.811+0000 7f6567200640 1 -- 172.21.15.84:0/2870983712 wait complete. 2024-04-28T13:06:39.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-28T13:06:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:39 smithi179 ceph-mon[63029]: pgmap v868: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:39 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/1463995460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:06:39.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:39 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/647824172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:06:39.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:39 smithi084 ceph-mon[28081]: pgmap v868: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:39.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:39 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/1463995460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:06:39.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:39 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/647824172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:06:39.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:39 smithi161 ceph-mon[62387]: pgmap v868: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-28T13:06:39.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:39 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/1463995460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-28T13:06:39.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:39 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/647824172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-28T13:06:40.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.031+0000 7f160b200640 1 -- 172.21.15.84:0/2220989120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f160c104310 msgr2=0x7f160c10aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:40.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.031+0000 7f160b200640 1 --2- 172.21.15.84:0/2220989120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f160c104310 0x7f160c10aba0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1604009ca0 tx=0x7f160402a8c0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.031+0000 7f160b200640 1 -- 172.21.15.84:0/2220989120 shutdown_connections 2024-04-28T13:06:40.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.031+0000 7f160b200640 1 --2- 172.21.15.84:0/2220989120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f160c104310 0x7f160c10aba0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.031+0000 7f160b200640 1 --2- 172.21.15.84:0/2220989120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f160c103950 0x7f160c103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.031+0000 7f160b200640 1 --2- 172.21.15.84:0/2220989120 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f160c102760 0x7f160c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.031+0000 7f160b200640 1 -- 172.21.15.84:0/2220989120 >> 172.21.15.84:0/2220989120 conn(0x7f160c0fdcb0 msgr2=0x7f160c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:40.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.032+0000 7f160b200640 1 -- 172.21.15.84:0/2220989120 shutdown_connections 2024-04-28T13:06:40.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.032+0000 7f160b200640 1 -- 172.21.15.84:0/2220989120 wait complete. 2024-04-28T13:06:40.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.032+0000 7f160b200640 1 Processor -- start 2024-04-28T13:06:40.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.033+0000 7f160b200640 1 -- start start 2024-04-28T13:06:40.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.033+0000 7f160b200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f160c102760 0x7f160c19ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:40.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.033+0000 7f160b200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f160c103950 0x7f160c19b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:40.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.033+0000 7f160b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f160c104310 0x7f160c1a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:40.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.033+0000 7f160b200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f160c10e2e0 con 0x7f160c103950 2024-04-28T13:06:40.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.033+0000 7f160b200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f160c10e160 con 0x7f160c104310 2024-04-28T13:06:40.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.033+0000 7f160b200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f160c10e460 con 0x7f160c102760 2024-04-28T13:06:40.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.033+0000 7f1601400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f160c103950 0x7f160c19b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:40.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.033+0000 7f1601400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f160c103950 0x7f160c19b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:46270/0 (socket says 172.21.15.84:46270) 2024-04-28T13:06:40.036 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.033+0000 7f1601400640 1 -- 172.21.15.84:0/66118461 learned_addr learned my addr 172.21.15.84:0/66118461 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:40.036 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.033+0000 7f1609e00640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f160c102760 0x7f160c19ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:40.036 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.033+0000 7f160a800640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f160c104310 0x7f160c1a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:40.036 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.034+0000 7f1601400640 1 -- 172.21.15.84:0/66118461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f160c102760 msgr2=0x7f160c19ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:40.036 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.034+0000 7f1601400640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f160c102760 0x7f160c19ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.036 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.034+0000 7f1601400640 1 -- 172.21.15.84:0/66118461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f160c104310 msgr2=0x7f160c1a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:40.036 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.034+0000 7f1601400640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f160c104310 0x7f160c1a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.036 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.034+0000 7f1601400640 1 -- 172.21.15.84:0/66118461 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f160c1a2760 con 0x7f160c103950 2024-04-28T13:06:40.036 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.034+0000 7f1601400640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f160c103950 0x7f160c19b080 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f15f400e9f0 tx=0x7f15f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:40.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.034+0000 7f1603400640 1 -- 172.21.15.84:0/66118461 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f400cd80 con 0x7f160c103950 2024-04-28T13:06:40.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.034+0000 7f160b200640 1 -- 172.21.15.84:0/66118461 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f160c1a2a50 con 0x7f160c103950 2024-04-28T13:06:40.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.034+0000 7f1603400640 1 -- 172.21.15.84:0/66118461 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15f40054d0 con 0x7f160c103950 2024-04-28T13:06:40.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.034+0000 7f1603400640 1 -- 172.21.15.84:0/66118461 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f4010690 con 0x7f160c103950 2024-04-28T13:06:40.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.035+0000 7f160b200640 1 -- 172.21.15.84:0/66118461 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f160c1a2f50 con 0x7f160c103950 2024-04-28T13:06:40.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.036+0000 7f160b200640 1 -- 172.21.15.84:0/66118461 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f160c1a31e0 con 0x7f160c103950 2024-04-28T13:06:40.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.038+0000 7f1603400640 1 -- 172.21.15.84:0/66118461 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f15f4004e90 con 0x7f160c103950 2024-04-28T13:06:40.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.039+0000 7f1603400640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f15f00a3c40 0x7f15f00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:40.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.039+0000 7f1603400640 1 -- 172.21.15.84:0/66118461 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f15f40d0d50 con 0x7f160c103950 2024-04-28T13:06:40.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.039+0000 7f1609e00640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f15f00a3c40 0x7f15f00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:40.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.040+0000 7f1609e00640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f15f00a3c40 0x7f15f00a6100 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f160c1037b0 tx=0x7f15fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:40.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.044+0000 7f1603400640 1 -- 172.21.15.84:0/66118461 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f15f409d1f0 con 0x7f160c103950 2024-04-28T13:06:40.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.258+0000 7f160b200640 1 -- 172.21.15.84:0/66118461 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f160c1a33b0 con 0x7f160c103950 2024-04-28T13:06:40.259 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.259+0000 7f1603400640 1 -- 172.21.15.84:0/66118461 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v94) ==== 81+0+591 (secure 0 0 0) 0x7f15f409caf0 con 0x7f160c103950 2024-04-28T13:06:40.259 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/1611938555 2024-04-29T12:40:29.445750+0000 2024-04-28T13:06:40.259 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:6801/4213070051 2024-04-29T12:40:29.445750+0000 2024-04-28T13:06:40.259 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:6800/4213070051 2024-04-29T12:40:29.445750+0000 2024-04-28T13:06:40.259 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/173746096 2024-04-29T12:40:29.445750+0000 2024-04-28T13:06:40.260 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/3031983413 2024-04-29T12:39:55.311912+0000 2024-04-28T13:06:40.260 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/3332561455 2024-04-29T12:39:55.311912+0000 2024-04-28T13:06:40.260 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/3243457783 2024-04-29T12:39:55.311912+0000 2024-04-28T13:06:40.260 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:6800/4148140420 2024-04-29T12:39:55.311912+0000 2024-04-28T13:06:40.260 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:0/3787967310 2024-04-29T12:40:29.445750+0000 2024-04-28T13:06:40.260 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84:6801/4148140420 2024-04-29T12:39:55.311912+0000 2024-04-28T13:06:40.260 INFO:teuthology.orchestra.run.smithi084.stderr:listed 10 entries 2024-04-28T13:06:40.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.262+0000 7f160b200640 1 -- 172.21.15.84:0/66118461 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f15f00a3c40 msgr2=0x7f15f00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:40.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.262+0000 7f160b200640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f15f00a3c40 0x7f15f00a6100 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f160c1037b0 tx=0x7f15fc005f90 comp rx=0 tx=0).stop 2024-04-28T13:06:40.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.262+0000 7f160b200640 1 -- 172.21.15.84:0/66118461 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f160c103950 msgr2=0x7f160c19b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:40.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.262+0000 7f160b200640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f160c103950 0x7f160c19b080 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f15f400e9f0 tx=0x7f15f400edf0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.262+0000 7f160b200640 1 -- 172.21.15.84:0/66118461 shutdown_connections 2024-04-28T13:06:40.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.262+0000 7f160b200640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f15f00a3c40 0x7f15f00a6100 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.262+0000 7f160b200640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f160c104310 0x7f160c1a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.262+0000 7f160b200640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f160c103950 0x7f160c19b080 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.262+0000 7f160b200640 1 --2- 172.21.15.84:0/66118461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f160c102760 0x7f160c19ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.262+0000 7f160b200640 1 -- 172.21.15.84:0/66118461 >> 172.21.15.84:0/66118461 conn(0x7f160c0fdcb0 msgr2=0x7f160c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:40.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.262+0000 7f160b200640 1 -- 172.21.15.84:0/66118461 shutdown_connections 2024-04-28T13:06:40.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.262+0000 7f160b200640 1 -- 172.21.15.84:0/66118461 wait complete. 2024-04-28T13:06:40.281 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-28T13:06:40.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:06:40.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-28T13:06:40.369 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi084.front.sepia.ceph.com 2024-04-28T13:06:40.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:06:40.369 DEBUG:teuthology.orchestra.run.smithi084:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-28T13:06:40.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:06:40.416 DEBUG:teuthology.orchestra.run.smithi084:> ip netns list 2024-04-28T13:06:40.513 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-28T13:06:40.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:06:40.514 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-28T13:06:40.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-28T13:06:40.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo ip link delete ceph-brx 2024-04-28T13:06:40.709 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-28T13:06:40.720 INFO:tasks.cephadm:Teardown begin 2024-04-28T13:06:40.720 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-28T13:06:40.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:06:40.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.903+0000 7fe2f3e00640 1 -- 172.21.15.84:0/2949001235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2f4100fe0 msgr2=0x7fe2f4103410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:40.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.903+0000 7fe2f3e00640 1 --2- 172.21.15.84:0/2949001235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2f4100fe0 0x7fe2f4103410 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4009900 tx=0x7fe2e402a3f0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.903+0000 7fe2f3e00640 1 -- 172.21.15.84:0/2949001235 shutdown_connections 2024-04-28T13:06:40.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.903+0000 7fe2f3e00640 1 --2- 172.21.15.84:0/2949001235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2f4106300 0x7fe2f4106780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.903+0000 7fe2f3e00640 1 --2- 172.21.15.84:0/2949001235 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2f4103950 0x7fe2f4105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.903+0000 7fe2f3e00640 1 --2- 172.21.15.84:0/2949001235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2f4100fe0 0x7fe2f4103410 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.903+0000 7fe2f3e00640 1 -- 172.21.15.84:0/2949001235 >> 172.21.15.84:0/2949001235 conn(0x7fe2f40fcc00 msgr2=0x7fe2f40ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:40.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.903+0000 7fe2f3e00640 1 -- 172.21.15.84:0/2949001235 shutdown_connections 2024-04-28T13:06:40.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.903+0000 7fe2f3e00640 1 -- 172.21.15.84:0/2949001235 wait complete. 2024-04-28T13:06:40.906 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.904+0000 7fe2f3e00640 1 Processor -- start 2024-04-28T13:06:40.906 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.904+0000 7fe2f3e00640 1 -- start start 2024-04-28T13:06:40.906 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.904+0000 7fe2f3e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2f4100fe0 0x7fe2f419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:40.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.904+0000 7fe2f3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2f4103950 0x7fe2f419f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:40.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.904+0000 7fe2f3e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2f4106300 0x7fe2f41a6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:40.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.904+0000 7fe2f3e00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f41148e0 con 0x7fe2f4106300 2024-04-28T13:06:40.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.904+0000 7fe2f3e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f4114760 con 0x7fe2f4103950 2024-04-28T13:06:40.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.904+0000 7fe2f3e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f4114a60 con 0x7fe2f4100fe0 2024-04-28T13:06:40.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.905+0000 7fe2f2000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2f4103950 0x7fe2f419f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:40.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.905+0000 7fe2f3400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2f4106300 0x7fe2f41a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:40.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.905+0000 7fe2f2000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2f4103950 0x7fe2f419f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:56276/0 (socket says 172.21.15.84:56276) 2024-04-28T13:06:40.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.905+0000 7fe2f3400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2f4106300 0x7fe2f41a6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:46286/0 (socket says 172.21.15.84:46286) 2024-04-28T13:06:40.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.905+0000 7fe2f2000640 1 -- 172.21.15.84:0/3896641841 learned_addr learned my addr 172.21.15.84:0/3896641841 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:40.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.905+0000 7fe2f2a00640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2f4100fe0 0x7fe2f419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:40.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.905+0000 7fe2f2000640 1 -- 172.21.15.84:0/3896641841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2f4100fe0 msgr2=0x7fe2f419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:40.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.905+0000 7fe2f2000640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2f4100fe0 0x7fe2f419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.905+0000 7fe2f2000640 1 -- 172.21.15.84:0/3896641841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2f4106300 msgr2=0x7fe2f41a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:40.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.905+0000 7fe2f2000640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2f4106300 0x7fe2f41a6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:40.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.905+0000 7fe2f2000640 1 -- 172.21.15.84:0/3896641841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f41a6bf0 con 0x7fe2f4103950 2024-04-28T13:06:40.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.905+0000 7fe2f3400640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2f4106300 0x7fe2f41a6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:06:40.908 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.906+0000 7fe2f2000640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2f4103950 0x7fe2f419f560 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000e9f0 tx=0x7fe2e000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:40.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.906+0000 7fe2eb400640 1 -- 172.21.15.84:0/3896641841 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e000cd80 con 0x7fe2f4103950 2024-04-28T13:06:40.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.906+0000 7fe2eb400640 1 -- 172.21.15.84:0/3896641841 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2e00054d0 con 0x7fe2f4103950 2024-04-28T13:06:40.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.906+0000 7fe2f3e00640 1 -- 172.21.15.84:0/3896641841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f41a6ee0 con 0x7fe2f4103950 2024-04-28T13:06:40.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.906+0000 7fe2eb400640 1 -- 172.21.15.84:0/3896641841 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0010690 con 0x7fe2f4103950 2024-04-28T13:06:40.909 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.906+0000 7fe2f3e00640 1 -- 172.21.15.84:0/3896641841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f41a73a0 con 0x7fe2f4103950 2024-04-28T13:06:40.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.907+0000 7fe2f3e00640 1 -- 172.21.15.84:0/3896641841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f40fe6f0 con 0x7fe2f4103950 2024-04-28T13:06:40.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.910+0000 7fe2eb400640 1 -- 172.21.15.84:0/3896641841 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe2e0004e90 con 0x7fe2f4103950 2024-04-28T13:06:40.910 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.911+0000 7fe2eb400640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe2d80a3cc0 0x7fe2d80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:40.911 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.911+0000 7fe2eb400640 1 -- 172.21.15.84:0/3896641841 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7fe2e00d0cd0 con 0x7fe2f4103950 2024-04-28T13:06:40.911 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.912+0000 7fe2f2a00640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe2d80a3cc0 0x7fe2d80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:40.911 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.912+0000 7fe2f2a00640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe2d80a3cc0 0x7fe2d80a6180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4031500 tx=0x7fe2e4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:40.915 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:40.916+0000 7fe2eb400640 1 -- 172.21.15.84:0/3896641841 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe2e009d170 con 0x7fe2f4103950 2024-04-28T13:06:40.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:40 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2870983712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:06:40.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:40 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/66118461' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:06:40.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:40 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2870983712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:06:40.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:40 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/66118461' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:06:40.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:40 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2870983712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-28T13:06:40.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:40 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/66118461' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-28T13:06:41.155 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.156+0000 7fe2f3e00640 1 -- 172.21.15.84:0/3896641841 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2f402dad0 con 0x7fe2f4103950 2024-04-28T13:06:41.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.157+0000 7fe2eb400640 1 -- 172.21.15.84:0/3896641841 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11884 (secure 0 0 0) 0x7fe2e009ca70 con 0x7fe2f4103950 2024-04-28T13:06:41.157 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T13:06:41.158 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6841/1196924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":1196924806},{"type":"v1","addr":"172.21.15.84:6841","nonce":1196924806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:34.334879+0000","modified":"2024-04-28T12:51:35.945937+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24562,"mds_2":24545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6839/3812720642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":3812720642},{"type":"v1","addr":"172.21.15.161:6839","nonce":3812720642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24602,24545,24562]},"id":1}]} 2024-04-28T13:06:41.158 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 25 2024-04-28T13:06:41.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.160+0000 7fe2f3e00640 1 -- 172.21.15.84:0/3896641841 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe2d80a3cc0 msgr2=0x7fe2d80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:41.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.160+0000 7fe2f3e00640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe2d80a3cc0 0x7fe2d80a6180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4031500 tx=0x7fe2e4005a20 comp rx=0 tx=0).stop 2024-04-28T13:06:41.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.160+0000 7fe2f3e00640 1 -- 172.21.15.84:0/3896641841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2f4103950 msgr2=0x7fe2f419f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:41.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.160+0000 7fe2f3e00640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2f4103950 0x7fe2f419f560 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000e9f0 tx=0x7fe2e000edf0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.160+0000 7fe2f3e00640 1 -- 172.21.15.84:0/3896641841 shutdown_connections 2024-04-28T13:06:41.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.160+0000 7fe2f3e00640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fe2d80a3cc0 0x7fe2d80a6180 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.160 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.160+0000 7fe2f3e00640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2f4106300 0x7fe2f41a6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.160+0000 7fe2f3e00640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2f4103950 0x7fe2f419f560 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.160+0000 7fe2f3e00640 1 --2- 172.21.15.84:0/3896641841 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe2f4100fe0 0x7fe2f419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.160+0000 7fe2f3e00640 1 -- 172.21.15.84:0/3896641841 >> 172.21.15.84:0/3896641841 conn(0x7fe2f40fcc00 msgr2=0x7fe2f4104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:41.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.160+0000 7fe2f3e00640 1 -- 172.21.15.84:0/3896641841 shutdown_connections 2024-04-28T13:06:41.161 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.160+0000 7fe2f3e00640 1 -- 172.21.15.84:0/3896641841 wait complete. 2024-04-28T13:06:41.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-28T13:06:41.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.388+0000 7ff152400640 1 -- 172.21.15.84:0/3723745302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff14c105d80 msgr2=0x7ff14c106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:41.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.388+0000 7ff152400640 1 --2- 172.21.15.84:0/3723745302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff14c105d80 0x7ff14c106200 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff134009900 tx=0x7ff13402a410 comp rx=0 tx=0).stop 2024-04-28T13:06:41.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.388+0000 7ff152400640 1 -- 172.21.15.84:0/3723745302 shutdown_connections 2024-04-28T13:06:41.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.388+0000 7ff152400640 1 --2- 172.21.15.84:0/3723745302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff14c106740 0x7ff14c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.388+0000 7ff152400640 1 --2- 172.21.15.84:0/3723745302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff14c105d80 0x7ff14c106200 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.388+0000 7ff152400640 1 --2- 172.21.15.84:0/3723745302 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff14c104b90 0x7ff14c104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.388+0000 7ff152400640 1 -- 172.21.15.84:0/3723745302 >> 172.21.15.84:0/3723745302 conn(0x7ff14c100120 msgr2=0x7ff14c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:41.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.388+0000 7ff152400640 1 -- 172.21.15.84:0/3723745302 shutdown_connections 2024-04-28T13:06:41.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.388+0000 7ff152400640 1 -- 172.21.15.84:0/3723745302 wait complete. 2024-04-28T13:06:41.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.389+0000 7ff152400640 1 Processor -- start 2024-04-28T13:06:41.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.389+0000 7ff152400640 1 -- start start 2024-04-28T13:06:41.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff152400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff14c104b90 0x7ff14c19ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:41.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff152400640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff14c105d80 0x7ff14c19b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:41.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff152400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff14c106740 0x7ff14c1a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:41.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff152400640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c1106a0 con 0x7ff14c106740 2024-04-28T13:06:41.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff152400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c110520 con 0x7ff14c104b90 2024-04-28T13:06:41.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff152400640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c110820 con 0x7ff14c105d80 2024-04-28T13:06:41.393 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff14be00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff14c106740 0x7ff14c1a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:41.393 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff14b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff14c104b90 0x7ff14c19ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:41.393 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff143e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff14c105d80 0x7ff14c19b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:41.393 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff14be00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff14c106740 0x7ff14c1a2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:46298/0 (socket says 172.21.15.84:46298) 2024-04-28T13:06:41.393 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff14b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff14c104b90 0x7ff14c19ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:56282/0 (socket says 172.21.15.84:56282) 2024-04-28T13:06:41.393 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff14be00640 1 -- 172.21.15.84:0/3580993457 learned_addr learned my addr 172.21.15.84:0/3580993457 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:41.393 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff14b400640 1 -- 172.21.15.84:0/3580993457 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff14c105d80 msgr2=0x7ff14c19b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:41.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff14b400640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff14c105d80 0x7ff14c19b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff14b400640 1 -- 172.21.15.84:0/3580993457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff14c106740 msgr2=0x7ff14c1a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:41.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff14b400640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff14c106740 0x7ff14c1a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff14b400640 1 -- 172.21.15.84:0/3580993457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14c1a2990 con 0x7ff14c104b90 2024-04-28T13:06:41.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.390+0000 7ff14be00640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff14c106740 0x7ff14c1a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:06:41.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.391+0000 7ff14b400640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff14c104b90 0x7ff14c19ad70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00e9c0 tx=0x7ff13c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:41.394 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.391+0000 7ff148c00640 1 -- 172.21.15.84:0/3580993457 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c00ccc0 con 0x7ff14c104b90 2024-04-28T13:06:41.395 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.391+0000 7ff152400640 1 -- 172.21.15.84:0/3580993457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff14c1a2c80 con 0x7ff14c104b90 2024-04-28T13:06:41.395 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.391+0000 7ff148c00640 1 -- 172.21.15.84:0/3580993457 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff13c0035f0 con 0x7ff14c104b90 2024-04-28T13:06:41.395 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.391+0000 7ff148c00640 1 -- 172.21.15.84:0/3580993457 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c00ce60 con 0x7ff14c104b90 2024-04-28T13:06:41.395 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.391+0000 7ff152400640 1 -- 172.21.15.84:0/3580993457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff14c1a31a0 con 0x7ff14c104b90 2024-04-28T13:06:41.395 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.392+0000 7ff152400640 1 -- 172.21.15.84:0/3580993457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff14c10bf30 con 0x7ff14c104b90 2024-04-28T13:06:41.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.395+0000 7ff148c00640 1 -- 172.21.15.84:0/3580993457 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff13c004000 con 0x7ff14c104b90 2024-04-28T13:06:41.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.396+0000 7ff148c00640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff1300a3c40 0x7ff1300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:41.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.396+0000 7ff148c00640 1 -- 172.21.15.84:0/3580993457 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff13c0d0e20 con 0x7ff14c104b90 2024-04-28T13:06:41.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.396+0000 7ff143e00640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff1300a3c40 0x7ff1300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:41.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.397+0000 7ff143e00640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff1300a3c40 0x7ff1300a6100 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff134000c00 tx=0x7ff1340314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:41.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.398+0000 7ff148c00640 1 -- 172.21.15.84:0/3580993457 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff13c09d350 con 0x7ff14c104b90 2024-04-28T13:06:41.638 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.639+0000 7ff152400640 1 -- 172.21.15.84:0/3580993457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff14c02dad0 con 0x7ff14c104b90 2024-04-28T13:06:41.639 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.640+0000 7ff148c00640 1 -- 172.21.15.84:0/3580993457 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11884 (secure 0 0 0) 0x7ff13c09cc50 con 0x7ff14c104b90 2024-04-28T13:06:41.640 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T13:06:41.640 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6835/3549575919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6834","nonce":3549575919},{"type":"v1","addr":"172.21.15.84:6835","nonce":3549575919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6837/2455857863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6836","nonce":2455857863},{"type":"v1","addr":"172.21.15.84:6837","nonce":2455857863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.84:6841/1196924806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6840","nonce":1196924806},{"type":"v1","addr":"172.21.15.84:6841","nonce":1196924806}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/2409757407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":2409757407},{"type":"v1","addr":"172.21.15.179:6833","nonce":2409757407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/3034002883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3034002883},{"type":"v1","addr":"172.21.15.161:6833","nonce":3034002883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/476402611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":476402611},{"type":"v1","addr":"172.21.15.179:6835","nonce":476402611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-28T12:48:34.334879+0000","modified":"2024-04-28T12:51:35.945937+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24562,"mds_2":24545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3348651842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3348651842},{"type":"v1","addr":"172.21.15.161:6835","nonce":3348651842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6837/2376856149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2376856149},{"type":"v1","addr":"172.21.15.179:6837","nonce":2376856149}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/3879704230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":3879704230},{"type":"v1","addr":"172.21.15.179:6839","nonce":3879704230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.161:6837/1346455857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1346455857},{"type":"v1","addr":"172.21.15.161:6837","nonce":1346455857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.84:6839/3772865132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6838","nonce":3772865132},{"type":"v1","addr":"172.21.15.84:6839","nonce":3772865132}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6839/3812720642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":3812720642},{"type":"v1","addr":"172.21.15.161:6839","nonce":3812720642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24602,24545,24562]},"id":1}]} 2024-04-28T13:06:41.641 INFO:teuthology.orchestra.run.smithi084.stderr:dumped fsmap epoch 25 2024-04-28T13:06:41.643 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.643+0000 7ff152400640 1 -- 172.21.15.84:0/3580993457 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff1300a3c40 msgr2=0x7ff1300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:41.643 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.643+0000 7ff152400640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff1300a3c40 0x7ff1300a6100 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff134000c00 tx=0x7ff1340314a0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.643 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.643+0000 7ff152400640 1 -- 172.21.15.84:0/3580993457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff14c104b90 msgr2=0x7ff14c19ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:41.643 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.643+0000 7ff152400640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff14c104b90 0x7ff14c19ad70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00e9c0 tx=0x7ff13c00ee80 comp rx=0 tx=0).stop 2024-04-28T13:06:41.643 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.643+0000 7ff152400640 1 -- 172.21.15.84:0/3580993457 shutdown_connections 2024-04-28T13:06:41.643 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.643+0000 7ff152400640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff1300a3c40 0x7ff1300a6100 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.644 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.643+0000 7ff152400640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff14c106740 0x7ff14c1a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.644 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.643+0000 7ff152400640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff14c105d80 0x7ff14c19b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.644 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.643+0000 7ff152400640 1 --2- 172.21.15.84:0/3580993457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff14c104b90 0x7ff14c19ad70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.644 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.643+0000 7ff152400640 1 -- 172.21.15.84:0/3580993457 >> 172.21.15.84:0/3580993457 conn(0x7ff14c100120 msgr2=0x7ff14c109100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:41.644 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.643+0000 7ff152400640 1 -- 172.21.15.84:0/3580993457 shutdown_connections 2024-04-28T13:06:41.644 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.643+0000 7ff152400640 1 -- 172.21.15.84:0/3580993457 wait complete. 2024-04-28T13:06:41.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-28T13:06:41.865 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.865+0000 7f1335000640 1 -- 172.21.15.84:0/1759922227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f133010a180 msgr2=0x7f133010a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:41.866 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.865+0000 7f1335000640 1 --2- 172.21.15.84:0/1759922227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f133010a180 0x7f133010a600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1318009860 tx=0x7f131802a340 comp rx=0 tx=0).stop 2024-04-28T13:06:41.866 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.865+0000 7f1335000640 1 -- 172.21.15.84:0/1759922227 shutdown_connections 2024-04-28T13:06:41.866 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.865+0000 7f1335000640 1 --2- 172.21.15.84:0/1759922227 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f133010ab40 0x7f13301113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.866 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.865+0000 7f1335000640 1 --2- 172.21.15.84:0/1759922227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f133010a180 0x7f133010a600 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.866 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.865+0000 7f1335000640 1 --2- 172.21.15.84:0/1759922227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1330108f90 0x7f13301093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.866 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.865+0000 7f1335000640 1 -- 172.21.15.84:0/1759922227 >> 172.21.15.84:0/1759922227 conn(0x7f1330077f40 msgr2=0x7f133007a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:41.866 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.866+0000 7f1335000640 1 -- 172.21.15.84:0/1759922227 shutdown_connections 2024-04-28T13:06:41.867 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.866+0000 7f1335000640 1 -- 172.21.15.84:0/1759922227 wait complete. 2024-04-28T13:06:41.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.867+0000 7f1335000640 1 Processor -- start 2024-04-28T13:06:41.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.867+0000 7f1335000640 1 -- start start 2024-04-28T13:06:41.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.867+0000 7f1335000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1330108f90 0x7f133019f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:41.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.867+0000 7f1335000640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f133010a180 0x7f133019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:41.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.867+0000 7f1335000640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f133010ab40 0x7f13301a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:41.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.867+0000 7f1335000640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f1330114ad0 con 0x7f133010ab40 2024-04-28T13:06:41.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.867+0000 7f1335000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1330114950 con 0x7f1330108f90 2024-04-28T13:06:41.869 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.867+0000 7f1335000640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f1330114c50 con 0x7f133010a180 2024-04-28T13:06:41.870 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.868+0000 7f132e800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f133010ab40 0x7f13301a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:41.870 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.868+0000 7f132de00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1330108f90 0x7f133019f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:41.870 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.868+0000 7f132e800640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f133010ab40 0x7f13301a6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:46310/0 (socket says 172.21.15.84:46310) 2024-04-28T13:06:41.870 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.868+0000 7f132e800640 1 -- 172.21.15.84:0/4182340393 learned_addr learned my addr 172.21.15.84:0/4182340393 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:41.870 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.868+0000 7f1327e00640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f133010a180 0x7f133019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:41.870 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.868+0000 7f132de00640 1 -- 172.21.15.84:0/4182340393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f133010a180 msgr2=0x7f133019f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:41.870 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.868+0000 7f132de00640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f133010a180 0x7f133019f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.870 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.868+0000 7f132de00640 1 -- 172.21.15.84:0/4182340393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f133010ab40 msgr2=0x7f13301a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:41.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.868+0000 7f132de00640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f133010ab40 0x7f13301a6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:41.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.868+0000 7f132de00640 1 -- 172.21.15.84:0/4182340393 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13301a6df0 con 0x7f1330108f90 2024-04-28T13:06:41.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.868+0000 7f132e800640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f133010ab40 0x7f13301a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:06:41.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.868+0000 7f132de00640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1330108f90 0x7f133019f1d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f132000e9c0 tx=0x7f132000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:41.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.869+0000 7f1326a00640 1 -- 172.21.15.84:0/4182340393 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132000ccc0 con 0x7f1330108f90 2024-04-28T13:06:41.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.869+0000 7f1335000640 1 -- 172.21.15.84:0/4182340393 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13301a70e0 con 0x7f1330108f90 2024-04-28T13:06:41.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.869+0000 7f1326a00640 1 -- 172.21.15.84:0/4182340393 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13200035f0 con 0x7f1330108f90 2024-04-28T13:06:41.871 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.869+0000 7f1326a00640 1 -- 172.21.15.84:0/4182340393 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132000ce60 con 0x7f1330108f90 2024-04-28T13:06:41.872 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.869+0000 7f1335000640 1 -- 172.21.15.84:0/4182340393 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13301a7600 con 0x7f1330108f90 2024-04-28T13:06:41.872 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.870+0000 7f1335000640 1 -- 172.21.15.84:0/4182340393 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13301102e0 con 0x7f1330108f90 2024-04-28T13:06:41.872 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.872+0000 7f1326a00640 1 -- 172.21.15.84:0/4182340393 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1320004000 con 0x7f1330108f90 2024-04-28T13:06:41.873 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.873+0000 7f1326a00640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f13140a3c40 0x7f13140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:41.873 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.873+0000 7f1326a00640 1 -- 172.21.15.84:0/4182340393 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7f13200d0da0 con 0x7f1330108f90 2024-04-28T13:06:41.873 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.874+0000 7f1327e00640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f13140a3c40 0x7f13140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:41.874 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.874+0000 7f1327e00640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f13140a3c40 0x7f13140a6100 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f131802a930 tx=0x7f1318030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:41.877 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:41.877+0000 7f1326a00640 1 -- 172.21.15.84:0/4182340393 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f132009d240 con 0x7f1330108f90 2024-04-28T13:06:41.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:41 smithi084 ceph-mon[28081]: pgmap v869: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:41.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:41 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3896641841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:06:41.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:41 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3580993457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:06:41.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:41 smithi161 ceph-mon[62387]: pgmap v869: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:41.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:41 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3896641841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:06:41.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:41 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3580993457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:06:42.082 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.082+0000 7f1335000640 1 -- 172.21.15.84:0/4182340393 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f133002dad0 con 0x7f1330108f90 2024-04-28T13:06:42.085 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.084+0000 7f1326a00640 1 -- 172.21.15.84:0/4182340393 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24634 (secure 0 0 0) 0x7f132009cb40 con 0x7f1330108f90 2024-04-28T13:06:42.085 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-28T13:06:42.085 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":94,"fsid":"ddee604a-055b-11ef-bc93-c7b262605968","created":"2024-04-28T12:39:22.248672+0000","modified":"2024-04-28T12:50:30.848698+0000","last_up_change":"2024-04-28T12:47:35.655536+0000","last_in_change":"2024-04-28T12:47:15.537594+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-28T12:43:15.608080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-28T12:48:32.037809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-28T12:48:33.240166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-28T12:48:37.567238+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2099232c-3140-41c4-83b4-89da6fa3a2fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6803","nonce":598031287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6805","nonce":598031287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6809","nonce":598031287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":598031287},{"type":"v1","addr":"172.21.15.84:6807","nonce":598031287}]},"public_addr":"172.21.15.84:6803/598031287","cluster_addr":"172.21.15.84:6805/598031287","heartbeat_back_addr":"172.21.15.84:6809/598031287","heartbeat_front_addr":"172.21.15.84:6807/598031287","state":["exists","up"]},{"osd":1,"uuid":"ee0ea48d-5933-4d73-a6ee-a9028badef12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6801","nonce":449593633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6803","nonce":449593633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6807","nonce":449593633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":449593633},{"type":"v1","addr":"172.21.15.161:6805","nonce":449593633}]},"public_addr":"172.21.15.161:6801/449593633","cluster_addr":"172.21.15.161:6803/449593633","heartbeat_back_addr":"172.21.15.161:6807/449593633","heartbeat_front_addr":"172.21.15.161:6805/449593633","state":["exists","up"]},{"osd":2,"uuid":"ac1f166c-e9a5-483c-9551-026ed03d1564","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6801","nonce":3974191146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6803","nonce":3974191146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6807","nonce":3974191146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":3974191146},{"type":"v1","addr":"172.21.15.179:6805","nonce":3974191146}]},"public_addr":"172.21.15.179:6801/3974191146","cluster_addr":"172.21.15.179:6803/3974191146","heartbeat_back_addr":"172.21.15.179:6807/3974191146","heartbeat_front_addr":"172.21.15.179:6805/3974191146","state":["exists","up"]},{"osd":3,"uuid":"547c4122-2dd9-4a63-a484-f615f99b428c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6811","nonce":3690923628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6813","nonce":3690923628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6817","nonce":3690923628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3690923628},{"type":"v1","addr":"172.21.15.84:6815","nonce":3690923628}]},"public_addr":"172.21.15.84:6811/3690923628","cluster_addr":"172.21.15.84:6813/3690923628","heartbeat_back_addr":"172.21.15.84:6817/3690923628","heartbeat_front_addr":"172.21.15.84:6815/3690923628","state":["exists","up"]},{"osd":4,"uuid":"dd99b566-f29a-42c6-b04d-7122b6787464","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6809","nonce":1673806515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6811","nonce":1673806515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6815","nonce":1673806515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":1673806515},{"type":"v1","addr":"172.21.15.161:6813","nonce":1673806515}]},"public_addr":"172.21.15.161:6809/1673806515","cluster_addr":"172.21.15.161:6811/1673806515","heartbeat_back_addr":"172.21.15.161:6815/1673806515","heartbeat_front_addr":"172.21.15.161:6813/1673806515","state":["exists","up"]},{"osd":5,"uuid":"e0697f1f-142e-43d7-b718-73f40970276e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6809","nonce":2387861251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6811","nonce":2387861251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6815","nonce":2387861251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":2387861251},{"type":"v1","addr":"172.21.15.179:6813","nonce":2387861251}]},"public_addr":"172.21.15.179:6809/2387861251","cluster_addr":"172.21.15.179:6811/2387861251","heartbeat_back_addr":"172.21.15.179:6815/2387861251","heartbeat_front_addr":"172.21.15.179:6813/2387861251","state":["exists","up"]},{"osd":6,"uuid":"1057554e-494f-4f9e-9823-a9e63c483a5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6819","nonce":440861785}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6821","nonce":440861785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6825","nonce":440861785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":440861785},{"type":"v1","addr":"172.21.15.84:6823","nonce":440861785}]},"public_addr":"172.21.15.84:6819/440861785","cluster_addr":"172.21.15.84:6821/440861785","heartbeat_back_addr":"172.21.15.84:6825/440861785","heartbeat_front_addr":"172.21.15.84:6823/440861785","state":["exists","up"]},{"osd":7,"uuid":"7108aa59-cf9a-4903-89fa-d0c456f0ed1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6817","nonce":1226878087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6819","nonce":1226878087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6823","nonce":1226878087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1226878087},{"type":"v1","addr":"172.21.15.161:6821","nonce":1226878087}]},"public_addr":"172.21.15.161:6817/1226878087","cluster_addr":"172.21.15.161:6819/1226878087","heartbeat_back_addr":"172.21.15.161:6823/1226878087","heartbeat_front_addr":"172.21.15.161:6821/1226878087","state":["exists","up"]},{"osd":8,"uuid":"0b005d60-667b-4061-80e9-af319ce052df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6817","nonce":277418948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6819","nonce":277418948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6823","nonce":277418948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":277418948},{"type":"v1","addr":"172.21.15.179:6821","nonce":277418948}]},"public_addr":"172.21.15.179:6817/277418948","cluster_addr":"172.21.15.179:6819/277418948","heartbeat_back_addr":"172.21.15.179:6823/277418948","heartbeat_front_addr":"172.21.15.179:6821/277418948","state":["exists","up"]},{"osd":9,"uuid":"7bc308bb-cc85-40b4-9af0-7dec273748f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6826","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6827","nonce":3448177312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6828","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6829","nonce":3448177312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6832","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6833","nonce":3448177312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6830","nonce":3448177312},{"type":"v1","addr":"172.21.15.84:6831","nonce":3448177312}]},"public_addr":"172.21.15.84:6827/3448177312","cluster_addr":"172.21.15.84:6829/3448177312","heartbeat_back_addr":"172.21.15.84:6833/3448177312","heartbeat_front_addr":"172.21.15.84:6831/3448177312","state":["exists","up"]},{"osd":10,"uuid":"3f25ff51-6d47-4313-a7af-4be37046c627","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6825","nonce":3240597178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6827","nonce":3240597178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6831","nonce":3240597178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3240597178},{"type":"v1","addr":"172.21.15.161:6829","nonce":3240597178}]},"public_addr":"172.21.15.161:6825/3240597178","cluster_addr":"172.21.15.161:6827/3240597178","heartbeat_back_addr":"172.21.15.161:6831/3240597178","heartbeat_front_addr":"172.21.15.161:6829/3240597178","state":["exists","up"]},{"osd":11,"uuid":"6babaf1e-dfe0-4f33-a484-ba66a566d5d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6825","nonce":3474527890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6827","nonce":3474527890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6831","nonce":3474527890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":3474527890},{"type":"v1","addr":"172.21.15.179:6829","nonce":3474527890}]},"public_addr":"172.21.15.179:6825/3474527890","cluster_addr":"172.21.15.179:6827/3474527890","heartbeat_back_addr":"172.21.15.179:6831/3474527890","heartbeat_front_addr":"172.21.15.179:6829/3474527890","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:22.992437+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:42:48.087571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:12.190096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:43:45.667280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:11.969802+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:44:37.412532+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:12.611189+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:45:39.429820+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:05.395459+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:46:40.142743+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:08.036605+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-28T12:47:33.880765+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.8","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1611938555":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6800/4213070051":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/173746096":"2024-04-29T12:40:29.445750+0000","172.21.15.84:0/3031983413":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3332561455":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3243457783":"2024-04-29T12:39:55.311912+0000","172.21.15.84:6800/4148140420":"2024-04-29T12:39:55.311912+0000","172.21.15.84:0/3787967310":"2024-04-29T12:40:29.445750+0000","172.21.15.84:6801/4148140420":"2024-04-29T12:39:55.311912+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-28T13:06:42.088 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.087+0000 7f1335000640 1 -- 172.21.15.84:0/4182340393 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f13140a3c40 msgr2=0x7f13140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:42.088 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.087+0000 7f1335000640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f13140a3c40 0x7f13140a6100 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f131802a930 tx=0x7f1318030450 comp rx=0 tx=0).stop 2024-04-28T13:06:42.088 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.087+0000 7f1335000640 1 -- 172.21.15.84:0/4182340393 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1330108f90 msgr2=0x7f133019f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:42.089 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.087+0000 7f1335000640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1330108f90 0x7f133019f1d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f132000e9c0 tx=0x7f132000ee80 comp rx=0 tx=0).stop 2024-04-28T13:06:42.089 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.087+0000 7f1335000640 1 -- 172.21.15.84:0/4182340393 shutdown_connections 2024-04-28T13:06:42.089 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.087+0000 7f1335000640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f13140a3c40 0x7f13140a6100 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.089 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.087+0000 7f1335000640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f133010ab40 0x7f13301a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.089 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.087+0000 7f1335000640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f133010a180 0x7f133019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.087+0000 7f1335000640 1 --2- 172.21.15.84:0/4182340393 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1330108f90 0x7f133019f1d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.087+0000 7f1335000640 1 -- 172.21.15.84:0/4182340393 >> 172.21.15.84:0/4182340393 conn(0x7f1330077f40 msgr2=0x7f133010d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:42.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.087+0000 7f1335000640 1 -- 172.21.15.84:0/4182340393 shutdown_connections 2024-04-28T13:06:42.090 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.087+0000 7f1335000640 1 -- 172.21.15.84:0/4182340393 wait complete. 2024-04-28T13:06:42.109 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-28T13:06:42.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:41 smithi179 ceph-mon[63029]: pgmap v869: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-28T13:06:42.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:41 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3896641841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:06:42.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:41 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3580993457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-28T13:06:42.308 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.307+0000 7ff56f600640 1 -- 172.21.15.84:0/3423549118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff568105d70 msgr2=0x7ff5681061f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:42.308 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.307+0000 7ff56f600640 1 --2- 172.21.15.84:0/3423549118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff568105d70 0x7ff5681061f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff558009900 tx=0x7ff55802a3f0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.307+0000 7ff56f600640 1 -- 172.21.15.84:0/3423549118 shutdown_connections 2024-04-28T13:06:42.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.307+0000 7ff56f600640 1 --2- 172.21.15.84:0/3423549118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff568106730 0x7ff56810cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.307+0000 7ff56f600640 1 --2- 172.21.15.84:0/3423549118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff568105d70 0x7ff5681061f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.307+0000 7ff56f600640 1 --2- 172.21.15.84:0/3423549118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff568104b80 0x7ff568104fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.307+0000 7ff56f600640 1 -- 172.21.15.84:0/3423549118 >> 172.21.15.84:0/3423549118 conn(0x7ff5681000f0 msgr2=0x7ff568102530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:42.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.307+0000 7ff56f600640 1 -- 172.21.15.84:0/3423549118 shutdown_connections 2024-04-28T13:06:42.309 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.307+0000 7ff56f600640 1 -- 172.21.15.84:0/3423549118 wait complete. 2024-04-28T13:06:42.312 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.308+0000 7ff56f600640 1 Processor -- start 2024-04-28T13:06:42.312 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.308+0000 7ff56f600640 1 -- start start 2024-04-28T13:06:42.312 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff56f600640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff568104b80 0x7ff56819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:42.312 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff56f600640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff568105d70 0x7ff56819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:42.312 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff56f600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff568106730 0x7ff5681a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:42.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff56f600640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7ff5681106e0 con 0x7ff568104b80 2024-04-28T13:06:42.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff56f600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff568110560 con 0x7ff568106730 2024-04-28T13:06:42.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff56f600640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7ff568110860 con 0x7ff568105d70 2024-04-28T13:06:42.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff56cc00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff568104b80 0x7ff56819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:42.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff56cc00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff568104b80 0x7ff56819adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:46332/0 (socket says 172.21.15.84:46332) 2024-04-28T13:06:42.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff56cc00640 1 -- 172.21.15.84:0/2745896504 learned_addr learned my addr 172.21.15.84:0/2745896504 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:42.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff567e00640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff568105d70 0x7ff56819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:42.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff56d600640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff568106730 0x7ff5681a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:42.313 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff56cc00640 1 -- 172.21.15.84:0/2745896504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff568105d70 msgr2=0x7ff56819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:42.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.309+0000 7ff56cc00640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff568105d70 0x7ff56819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.310+0000 7ff56cc00640 1 -- 172.21.15.84:0/2745896504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff568106730 msgr2=0x7ff5681a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:42.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.310+0000 7ff56cc00640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff568106730 0x7ff5681a2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.310+0000 7ff56cc00640 1 -- 172.21.15.84:0/2745896504 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5681a29d0 con 0x7ff568104b80 2024-04-28T13:06:42.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.310+0000 7ff56cc00640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff568104b80 0x7ff56819adb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff55c00e9c0 tx=0x7ff55c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:42.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.310+0000 7ff565600640 1 -- 172.21.15.84:0/2745896504 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c00ccc0 con 0x7ff568104b80 2024-04-28T13:06:42.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.310+0000 7ff565600640 1 -- 172.21.15.84:0/2745896504 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff55c0035f0 con 0x7ff568104b80 2024-04-28T13:06:42.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.310+0000 7ff56f600640 1 -- 172.21.15.84:0/2745896504 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5681a2cc0 con 0x7ff568104b80 2024-04-28T13:06:42.314 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.310+0000 7ff565600640 1 -- 172.21.15.84:0/2745896504 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c00ce60 con 0x7ff568104b80 2024-04-28T13:06:42.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.310+0000 7ff56f600640 1 -- 172.21.15.84:0/2745896504 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5681a3190 con 0x7ff568104b80 2024-04-28T13:06:42.315 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.311+0000 7ff556a00640 1 -- 172.21.15.84:0/2745896504 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff568063530 con 0x7ff568104b80 2024-04-28T13:06:42.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.313+0000 7ff565600640 1 -- 172.21.15.84:0/2745896504 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff55c0037a0 con 0x7ff568104b80 2024-04-28T13:06:42.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.316+0000 7ff565600640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff5500a3c40 0x7ff5500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:42.316 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.316+0000 7ff565600640 1 -- 172.21.15.84:0/2745896504 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9121+0+0 (secure 0 0 0) 0x7ff55c0d0cc0 con 0x7ff568104b80 2024-04-28T13:06:42.317 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.316+0000 7ff567e00640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff5500a3c40 0x7ff5500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:42.318 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.317+0000 7ff567e00640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff5500a3c40 0x7ff5500a6100 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff558031500 tx=0x7ff558005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:42.320 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.319+0000 7ff565600640 1 -- 172.21.15.84:0/2745896504 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff55c09d1c0 con 0x7ff568104b80 2024-04-28T13:06:42.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.561+0000 7ff556a00640 1 -- 172.21.15.84:0/2745896504 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff568063700 con 0x7ff568104b80 2024-04-28T13:06:42.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.566+0000 7ff565600640 1 -- 172.21.15.84:0/2745896504 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7ff55c09cac0 con 0x7ff568104b80 2024-04-28T13:06:42.566 INFO:teuthology.orchestra.run.smithi084.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-28T13:06:42.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.569+0000 7ff556a00640 1 -- 172.21.15.84:0/2745896504 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff5500a3c40 msgr2=0x7ff5500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:42.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.569+0000 7ff556a00640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff5500a3c40 0x7ff5500a6100 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff558031500 tx=0x7ff558005a20 comp rx=0 tx=0).stop 2024-04-28T13:06:42.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.569+0000 7ff556a00640 1 -- 172.21.15.84:0/2745896504 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff568104b80 msgr2=0x7ff56819adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:42.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.569+0000 7ff556a00640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff568104b80 0x7ff56819adb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff55c00e9c0 tx=0x7ff55c00edc0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.569+0000 7ff556a00640 1 -- 172.21.15.84:0/2745896504 shutdown_connections 2024-04-28T13:06:42.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.569+0000 7ff556a00640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7ff5500a3c40 0x7ff5500a6100 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.569+0000 7ff556a00640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff568106730 0x7ff5681a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.571 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.569+0000 7ff556a00640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff568105d70 0x7ff56819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.571 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.569+0000 7ff556a00640 1 --2- 172.21.15.84:0/2745896504 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff568104b80 0x7ff56819adb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.571 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.569+0000 7ff556a00640 1 -- 172.21.15.84:0/2745896504 >> 172.21.15.84:0/2745896504 conn(0x7ff5681000f0 msgr2=0x7ff568109140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:42.571 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.569+0000 7ff556a00640 1 -- 172.21.15.84:0/2745896504 shutdown_connections 2024-04-28T13:06:42.571 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.569+0000 7ff556a00640 1 -- 172.21.15.84:0/2745896504 wait complete. 2024-04-28T13:06:42.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-28T13:06:42.763 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.762+0000 7f7066e00640 1 -- 172.21.15.84:0/141456576 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7060105da0 msgr2=0x7f7060106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:42.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.762+0000 7f7066e00640 1 --2- 172.21.15.84:0/141456576 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7060105da0 0x7f7060106220 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f704c009900 tx=0x7f704c02a3f0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.763+0000 7f7066e00640 1 -- 172.21.15.84:0/141456576 shutdown_connections 2024-04-28T13:06:42.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.763+0000 7f7066e00640 1 --2- 172.21.15.84:0/141456576 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7060106760 0x7f706010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.763+0000 7f7066e00640 1 --2- 172.21.15.84:0/141456576 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7060105da0 0x7f7060106220 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.763+0000 7f7066e00640 1 --2- 172.21.15.84:0/141456576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7060104bb0 0x7f7060104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.763+0000 7f7066e00640 1 -- 172.21.15.84:0/141456576 >> 172.21.15.84:0/141456576 conn(0x7f7060100120 msgr2=0x7f7060102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:42.764 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.763+0000 7f7066e00640 1 -- 172.21.15.84:0/141456576 shutdown_connections 2024-04-28T13:06:42.765 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.763+0000 7f7066e00640 1 -- 172.21.15.84:0/141456576 wait complete. 2024-04-28T13:06:42.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.764+0000 7f7066e00640 1 Processor -- start 2024-04-28T13:06:42.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.764+0000 7f7066e00640 1 -- start start 2024-04-28T13:06:42.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.764+0000 7f7066e00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7060104bb0 0x7f706019add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:42.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.764+0000 7f7066e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7060105da0 0x7f706019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:42.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.764+0000 7f7066e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7060106760 0x7f70601a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:42.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.764+0000 7f7066e00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f7060110730 con 0x7f7060105da0 2024-04-28T13:06:42.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.764+0000 7f7066e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f70601105b0 con 0x7f7060106760 2024-04-28T13:06:42.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.764+0000 7f7066e00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f70601108b0 con 0x7f7060104bb0 2024-04-28T13:06:42.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.764+0000 7f705f400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7060105da0 0x7f706019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:42.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.764+0000 7f705f400640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7060105da0 0x7f706019b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:46376/0 (socket says 172.21.15.84:46376) 2024-04-28T13:06:42.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.764+0000 7f705f400640 1 -- 172.21.15.84:0/4040774353 learned_addr learned my addr 172.21.15.84:0/4040774353 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:42.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.764+0000 7f705fe00640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7060104bb0 0x7f706019add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:42.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.765+0000 7f7064e00640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7060106760 0x7f70601a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:42.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.765+0000 7f705fe00640 1 -- 172.21.15.84:0/4040774353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7060106760 msgr2=0x7f70601a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:42.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.765+0000 7f705fe00640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7060106760 0x7f70601a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.765+0000 7f705fe00640 1 -- 172.21.15.84:0/4040774353 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7060105da0 msgr2=0x7f706019b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:42.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.765+0000 7f705fe00640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7060105da0 0x7f706019b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:42.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.765+0000 7f705fe00640 1 -- 172.21.15.84:0/4040774353 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70601a29f0 con 0x7f7060104bb0 2024-04-28T13:06:42.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.765+0000 7f705fe00640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7060104bb0 0x7f706019add0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f705000e9c0 tx=0x7f705000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:42.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.765+0000 7f705cc00640 1 -- 172.21.15.84:0/4040774353 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705000ccc0 con 0x7f7060104bb0 2024-04-28T13:06:42.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.765+0000 7f7066e00640 1 -- 172.21.15.84:0/4040774353 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70601a2ce0 con 0x7f7060104bb0 2024-04-28T13:06:42.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.766+0000 7f705cc00640 1 -- 172.21.15.84:0/4040774353 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70500044d0 con 0x7f7060104bb0 2024-04-28T13:06:42.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.766+0000 7f7066e00640 1 -- 172.21.15.84:0/4040774353 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70601a31d0 con 0x7f7060104bb0 2024-04-28T13:06:42.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.766+0000 7f705cc00640 1 -- 172.21.15.84:0/4040774353 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7050003e60 con 0x7f7060104bb0 2024-04-28T13:06:42.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.767+0000 7f7066e00640 1 -- 172.21.15.84:0/4040774353 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f706010bed0 con 0x7f7060104bb0 2024-04-28T13:06:42.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.769+0000 7f705cc00640 1 -- 172.21.15.84:0/4040774353 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f70500035f0 con 0x7f7060104bb0 2024-04-28T13:06:42.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.770+0000 7f705cc00640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f70440a3cc0 0x7f70440a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:42.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.770+0000 7f705cc00640 1 -- 172.21.15.84:0/4040774353 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9379+0+0 (secure 0 0 0) 0x7f70500d0da0 con 0x7f7060104bb0 2024-04-28T13:06:42.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.770+0000 7f705f400640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f70440a3cc0 0x7f70440a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:42.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.771+0000 7f705f400640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f70440a3cc0 0x7f70440a6180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f706019c2e0 tx=0x7f704c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:42.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:42.774+0000 7f705cc00640 1 -- 172.21.15.84:0/4040774353 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f705009d140 con 0x7f7060104bb0 2024-04-28T13:06:42.785 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 13:06:42 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g[66128]: ignoring --setuser ceph since I am not root 2024-04-28T13:06:42.785 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 13:06:42 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g[66128]: ignoring --setgroup ceph since I am not root 2024-04-28T13:06:42.785 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 13:06:42 smithi084 ceph-mds[66132]: -- 172.21.15.84:0/1011785082 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aabee3fa40 con 0x55aabee13000 2024-04-28T13:06:42.785 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 13:06:42 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g[66128]: starting mds.g at 2024-04-28T13:06:42.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:42 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a[28077]: 2024-04-28T13:06:42.562+0000 7f0896600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-28T13:06:42.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:42 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4182340393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:06:42.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:42 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2745896504' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-28T13:06:42.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:42 smithi084 ceph-mon[28081]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-28T13:06:42.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:42 smithi084 ceph-mon[28081]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-28T13:06:42.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:42 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/2745896504' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-28T13:06:42.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:42 smithi084 ceph-mon[28081]: osdmap e95: 12 total, 12 up, 12 in 2024-04-28T13:06:42.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:42 smithi084 ceph-mon[28081]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-28T13:06:42.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4182340393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:06:42.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2745896504' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-28T13:06:42.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-mon[63029]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-28T13:06:42.919 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-mon[63029]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-28T13:06:42.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/2745896504' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-28T13:06:42.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-mon[63029]: osdmap e95: 12 total, 12 up, 12 in 2024-04-28T13:06:42.920 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-mon[63029]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-28T13:06:42.920 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-i[94029]: ignoring --setuser ceph since I am not root 2024-04-28T13:06:42.920 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-i[94029]: ignoring --setgroup ceph since I am not root 2024-04-28T13:06:42.920 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-mds[94033]: -- 172.21.15.179:0/3793838916 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b2e5f33a40 con 0x55b2e5f06800 2024-04-28T13:06:42.920 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-i[94029]: starting mds.i at 2024-04-28T13:06:42.921 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-c[93721]: ignoring --setuser ceph since I am not root 2024-04-28T13:06:42.921 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-c[93721]: ignoring --setgroup ceph since I am not root 2024-04-28T13:06:42.921 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-mds[93725]: -- 172.21.15.179:0/2276487156 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5619c4869a40 con 0x5619c483d000 2024-04-28T13:06:42.921 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 13:06:42 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-c[93721]: starting mds.c at 2024-04-28T13:06:42.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4182340393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-28T13:06:42.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2745896504' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-28T13:06:42.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-mon[62387]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-28T13:06:42.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-mon[62387]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-28T13:06:42.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/2745896504' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-28T13:06:42.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-mon[62387]: osdmap e95: 12 total, 12 up, 12 in 2024-04-28T13:06:42.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-mon[62387]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-28T13:06:42.998 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b[92461]: ignoring --setuser ceph since I am not root 2024-04-28T13:06:42.998 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b[92461]: ignoring --setgroup ceph since I am not root 2024-04-28T13:06:42.998 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-mds[92465]: -- 172.21.15.161:0/1891561894 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ad113dba40 con 0x55ad113af000 2024-04-28T13:06:42.998 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b[92461]: starting mds.b at 2024-04-28T13:06:42.998 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-e[91844]: ignoring --setuser ceph since I am not root 2024-04-28T13:06:42.999 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-e[91844]: ignoring --setgroup ceph since I am not root 2024-04-28T13:06:42.999 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-mds[91848]: -- 172.21.15.161:0/985521814 <== mon.2 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563a4f6a7a40 con 0x563a4f67a800 2024-04-28T13:06:42.999 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-e[91844]: starting mds.e at 2024-04-28T13:06:42.999 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h[92154]: ignoring --setuser ceph since I am not root 2024-04-28T13:06:42.999 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h[92154]: ignoring --setgroup ceph since I am not root 2024-04-28T13:06:42.999 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-mds[92158]: -- 172.21.15.161:0/3372372900 <== mon.2 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a02f2a5a40 con 0x55a02f278800 2024-04-28T13:06:43.000 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 13:06:42 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h[92154]: starting mds.h at 2024-04-28T13:06:43.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.017+0000 7f7066e00640 1 -- 172.21.15.84:0/4040774353 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f70600635d0 con 0x7f7060104bb0 2024-04-28T13:06:43.574 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.573+0000 7f705cc00640 1 -- 172.21.15.84:0/4040774353 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7f705009ca40 con 0x7f7060104bb0 2024-04-28T13:06:43.578 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.576+0000 7f7066e00640 1 -- 172.21.15.84:0/4040774353 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f70440a3cc0 msgr2=0x7f70440a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:43.578 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.577+0000 7f7066e00640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f70440a3cc0 0x7f70440a6180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f706019c2e0 tx=0x7f704c005a20 comp rx=0 tx=0).stop 2024-04-28T13:06:43.578 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.577+0000 7f7066e00640 1 -- 172.21.15.84:0/4040774353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7060104bb0 msgr2=0x7f706019add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:43.578 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.577+0000 7f7066e00640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7060104bb0 0x7f706019add0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f705000e9c0 tx=0x7f705000ee80 comp rx=0 tx=0).stop 2024-04-28T13:06:43.578 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.577+0000 7f7066e00640 1 -- 172.21.15.84:0/4040774353 shutdown_connections 2024-04-28T13:06:43.578 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.577+0000 7f7066e00640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f70440a3cc0 0x7f70440a6180 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:43.578 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.577+0000 7f7066e00640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7060106760 0x7f70601a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:43.578 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.577+0000 7f7066e00640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7060105da0 0x7f706019b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:43.579 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.577+0000 7f7066e00640 1 --2- 172.21.15.84:0/4040774353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7060104bb0 0x7f706019add0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:43.579 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.577+0000 7f7066e00640 1 -- 172.21.15.84:0/4040774353 >> 172.21.15.84:0/4040774353 conn(0x7f7060100120 msgr2=0x7f70601090a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:43.579 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.577+0000 7f7066e00640 1 -- 172.21.15.84:0/4040774353 shutdown_connections 2024-04-28T13:06:43.579 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.577+0000 7f7066e00640 1 -- 172.21.15.84:0/4040774353 wait complete. 2024-04-28T13:06:43.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-28T13:06:43.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.771+0000 7fac3a200640 1 -- 172.21.15.84:0/2826947599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34103400 msgr2=0x7fac34109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:43.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.771+0000 7fac3a200640 1 --2- 172.21.15.84:0/2826947599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34103400 0x7fac34109c90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009900 tx=0x7fac2c02a3f0 comp rx=0 tx=0).stop 2024-04-28T13:06:43.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.771+0000 7fac3a200640 1 -- 172.21.15.84:0/2826947599 shutdown_connections 2024-04-28T13:06:43.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.771+0000 7fac3a200640 1 --2- 172.21.15.84:0/2826947599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34103400 0x7fac34109c90 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:43.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.771+0000 7fac3a200640 1 --2- 172.21.15.84:0/2826947599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34102a40 0x7fac34102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:43.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.771+0000 7fac3a200640 1 --2- 172.21.15.84:0/2826947599 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34101850 0x7fac34101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:43.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.771+0000 7fac3a200640 1 -- 172.21.15.84:0/2826947599 >> 172.21.15.84:0/2826947599 conn(0x7fac340fcdc0 msgr2=0x7fac340ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:43.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.771+0000 7fac3a200640 1 -- 172.21.15.84:0/2826947599 shutdown_connections 2024-04-28T13:06:43.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.771+0000 7fac3a200640 1 -- 172.21.15.84:0/2826947599 wait complete. 2024-04-28T13:06:43.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.772+0000 7fac3a200640 1 Processor -- start 2024-04-28T13:06:43.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.772+0000 7fac3a200640 1 -- start start 2024-04-28T13:06:43.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.772+0000 7fac3a200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34101850 0x7fac3410c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:43.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.772+0000 7fac3a200640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34102a40 0x7fac3410a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:43.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac3a200640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34103400 0x7fac3410ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:43.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac3a200640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7fac3410e500 con 0x7fac34102a40 2024-04-28T13:06:43.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac3a200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fac3410e380 con 0x7fac34101850 2024-04-28T13:06:43.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac3a200640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7fac3410e680 con 0x7fac34103400 2024-04-28T13:06:43.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac32a00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34102a40 0x7fac3410a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:43.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac33400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34101850 0x7fac3410c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:43.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac33400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34101850 0x7fac3410c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.84:56360/0 (socket says 172.21.15.84:56360) 2024-04-28T13:06:43.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac33400640 1 -- 172.21.15.84:0/3796225270 learned_addr learned my addr 172.21.15.84:0/3796225270 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:43.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac33e00640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34103400 0x7fac3410ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:43.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac33400640 1 -- 172.21.15.84:0/3796225270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34103400 msgr2=0x7fac3410ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:43.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac33400640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34103400 0x7fac3410ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:43.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac33400640 1 -- 172.21.15.84:0/3796225270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34102a40 msgr2=0x7fac3410a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:43.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac33400640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34102a40 0x7fac3410a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:43.778 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac33400640 1 -- 172.21.15.84:0/3796225270 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3410b5e0 con 0x7fac34101850 2024-04-28T13:06:43.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.773+0000 7fac32a00640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34102a40 0x7fac3410a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-28T13:06:43.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.774+0000 7fac33400640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34101850 0x7fac3410c1d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fac2400ea80 tx=0x7fac2400ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:43.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.774+0000 7fac2be00640 1 -- 172.21.15.84:0/3796225270 <== mon.2 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2400ce40 con 0x7fac34101850 2024-04-28T13:06:43.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.774+0000 7fac2be00640 1 -- 172.21.15.84:0/3796225270 <== mon.2 v2:172.21.15.161:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac24011920 con 0x7fac34101850 2024-04-28T13:06:43.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.774+0000 7fac3a200640 1 -- 172.21.15.84:0/3796225270 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac341a2c20 con 0x7fac34101850 2024-04-28T13:06:43.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.774+0000 7fac2be00640 1 -- 172.21.15.84:0/3796225270 <== mon.2 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2401aaa0 con 0x7fac34101850 2024-04-28T13:06:43.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.774+0000 7fac3a200640 1 -- 172.21.15.84:0/3796225270 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac341a3170 con 0x7fac34101850 2024-04-28T13:06:43.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.775+0000 7fac3a200640 1 -- 172.21.15.84:0/3796225270 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac34108b70 con 0x7fac34101850 2024-04-28T13:06:43.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.777+0000 7fac2be00640 1 -- 172.21.15.84:0/3796225270 <== mon.2 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fac2401ac40 con 0x7fac34101850 2024-04-28T13:06:43.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.778+0000 7fac2be00640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fac180a3c40 0x7fac180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:43.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.779+0000 7fac2be00640 1 -- 172.21.15.84:0/3796225270 <== mon.2 v2:172.21.15.161:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9379+0+0 (secure 0 0 0) 0x7fac240d0fe0 con 0x7fac34101850 2024-04-28T13:06:43.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.779+0000 7fac32a00640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fac180a3c40 0x7fac180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:43.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.779+0000 7fac32a00640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fac180a3c40 0x7fac180a6100 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fac3410bf40 tx=0x7fac1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:43.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.783+0000 7fac2be00640 1 -- 172.21.15.84:0/3796225270 <== mon.2 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fac24016030 con 0x7fac34101850 2024-04-28T13:06:43.922 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: pgmap v870: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:43.922 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/4040774353' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:06:43.922 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:06:43.922 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-28T13:06:43.922 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-28T13:06:43.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: Cluster is now healthy 2024-04-28T13:06:43.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-28T13:06:43.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.161:6834/4135570209,v1:172.21.15.161:6835/4135570209] up:boot 2024-04-28T13:06:43.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.179:6836/1391902342,v1:172.21.15.179:6838/1391902342] up:boot 2024-04-28T13:06:43.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.179:6837/4228448907,v1:172.21.15.179:6839/4228448907] up:boot 2024-04-28T13:06:43.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.161:6836/90405389,v1:172.21.15.161:6838/90405389] up:boot 2024-04-28T13:06:43.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.161:6837/1334566592,v1:172.21.15.161:6839/1334566592] up:boot 2024-04-28T13:06:43.923 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: mds.? [v2:172.21.15.84:6838/784964266,v1:172.21.15.84:6839/784964266] up:boot 2024-04-28T13:06:43.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: fsmap 12 up:standby 2024-04-28T13:06:43.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-28T13:06:43.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T13:06:43.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T13:06:43.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T13:06:43.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T13:06:43.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:43 smithi084 ceph-mon[28081]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T13:06:43.976 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:43.975+0000 7fac3a200640 1 -- 172.21.15.84:0/3796225270 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fac3402dad0 con 0x7fac34101850 2024-04-28T13:06:43.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: pgmap v870: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:43.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/4040774353' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:06:43.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:06:43.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-28T13:06:43.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-28T13:06:43.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: Cluster is now healthy 2024-04-28T13:06:43.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-28T13:06:43.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.161:6834/4135570209,v1:172.21.15.161:6835/4135570209] up:boot 2024-04-28T13:06:43.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.179:6836/1391902342,v1:172.21.15.179:6838/1391902342] up:boot 2024-04-28T13:06:43.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.179:6837/4228448907,v1:172.21.15.179:6839/4228448907] up:boot 2024-04-28T13:06:43.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.161:6836/90405389,v1:172.21.15.161:6838/90405389] up:boot 2024-04-28T13:06:43.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.161:6837/1334566592,v1:172.21.15.161:6839/1334566592] up:boot 2024-04-28T13:06:43.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: mds.? [v2:172.21.15.84:6838/784964266,v1:172.21.15.84:6839/784964266] up:boot 2024-04-28T13:06:43.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: fsmap 12 up:standby 2024-04-28T13:06:43.998 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-28T13:06:43.999 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T13:06:43.999 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T13:06:43.999 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T13:06:43.999 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T13:06:43.999 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:43 smithi161 ceph-mon[62387]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T13:06:44.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: pgmap v870: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-28T13:06:44.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/4040774353' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:06:44.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-28T13:06:44.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-28T13:06:44.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-28T13:06:44.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: Cluster is now healthy 2024-04-28T13:06:44.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-28T13:06:44.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.161:6834/4135570209,v1:172.21.15.161:6835/4135570209] up:boot 2024-04-28T13:06:44.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.179:6836/1391902342,v1:172.21.15.179:6838/1391902342] up:boot 2024-04-28T13:06:44.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.179:6837/4228448907,v1:172.21.15.179:6839/4228448907] up:boot 2024-04-28T13:06:44.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.161:6836/90405389,v1:172.21.15.161:6838/90405389] up:boot 2024-04-28T13:06:44.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.161:6837/1334566592,v1:172.21.15.161:6839/1334566592] up:boot 2024-04-28T13:06:44.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: mds.? [v2:172.21.15.84:6838/784964266,v1:172.21.15.84:6839/784964266] up:boot 2024-04-28T13:06:44.171 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: fsmap 12 up:standby 2024-04-28T13:06:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-28T13:06:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-28T13:06:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-28T13:06:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-28T13:06:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-28T13:06:44.172 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:43 smithi179 ceph-mon[63029]: from='mgr.14152 172.21.15.84:0/1031573102' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-28T13:06:44.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.702+0000 7fac2be00640 1 -- 172.21.15.84:0/3796225270 <== mon.2 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v96) ==== 181+0+0 (secure 0 0 0) 0x7fac2409d3a0 con 0x7fac34101850 2024-04-28T13:06:44.707 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'cephfs_metadata' removed 2024-04-28T13:06:44.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.707+0000 7fac3a200640 1 -- 172.21.15.84:0/3796225270 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fac180a3c40 msgr2=0x7fac180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:44.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.707+0000 7fac3a200640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fac180a3c40 0x7fac180a6100 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fac3410bf40 tx=0x7fac1c005f90 comp rx=0 tx=0).stop 2024-04-28T13:06:44.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.707+0000 7fac3a200640 1 -- 172.21.15.84:0/3796225270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34101850 msgr2=0x7fac3410c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:44.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.707+0000 7fac3a200640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34101850 0x7fac3410c1d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fac2400ea80 tx=0x7fac2400ef40 comp rx=0 tx=0).stop 2024-04-28T13:06:44.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.708+0000 7fac3a200640 1 -- 172.21.15.84:0/3796225270 shutdown_connections 2024-04-28T13:06:44.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.708+0000 7fac3a200640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7fac180a3c40 0x7fac180a6100 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:44.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.708+0000 7fac3a200640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac34103400 0x7fac3410ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:44.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.708+0000 7fac3a200640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac34102a40 0x7fac3410a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:44.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.708+0000 7fac3a200640 1 --2- 172.21.15.84:0/3796225270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac34101850 0x7fac3410c1d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:44.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.708+0000 7fac3a200640 1 -- 172.21.15.84:0/3796225270 >> 172.21.15.84:0/3796225270 conn(0x7fac340fcdc0 msgr2=0x7fac34105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:44.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.708+0000 7fac3a200640 1 -- 172.21.15.84:0/3796225270 shutdown_connections 2024-04-28T13:06:44.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.708+0000 7fac3a200640 1 -- 172.21.15.84:0/3796225270 wait complete. 2024-04-28T13:06:44.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-28T13:06:44.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.923+0000 7f817be00640 1 -- 172.21.15.84:0/1223378476 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f817c078ed0 msgr2=0x7f817c079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:44.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.923+0000 7f817be00640 1 --2- 172.21.15.84:0/1223378476 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f817c078ed0 0x7f817c079770 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8174009ca0 tx=0x7f817402a8b0 comp rx=0 tx=0).stop 2024-04-28T13:06:44.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.923+0000 7f817be00640 1 -- 172.21.15.84:0/1223378476 shutdown_connections 2024-04-28T13:06:44.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.923+0000 7f817be00640 1 --2- 172.21.15.84:0/1223378476 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f817c078ed0 0x7f817c079770 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:44.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.923+0000 7f817be00640 1 --2- 172.21.15.84:0/1223378476 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f817c078530 0x7f817c078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:44.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.923+0000 7f817be00640 1 --2- 172.21.15.84:0/1223378476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f817c0772f0 0x7f817c077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:44.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.923+0000 7f817be00640 1 -- 172.21.15.84:0/1223378476 >> 172.21.15.84:0/1223378476 conn(0x7f817c0fffe0 msgr2=0x7f817c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:44.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.923+0000 7f817be00640 1 -- 172.21.15.84:0/1223378476 shutdown_connections 2024-04-28T13:06:44.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.923+0000 7f817be00640 1 -- 172.21.15.84:0/1223378476 wait complete. 2024-04-28T13:06:44.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.924+0000 7f817be00640 1 Processor -- start 2024-04-28T13:06:44.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.925+0000 7f817be00640 1 -- start start 2024-04-28T13:06:44.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.925+0000 7f817be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f817c0772f0 0x7f817c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:44.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.925+0000 7f817be00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f817c078530 0x7f817c19f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:44.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.925+0000 7f817be00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f817c078ed0 0x7f817c1a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:44.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.925+0000 7f817be00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f817c1149f0 con 0x7f817c078530 2024-04-28T13:06:44.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.925+0000 7f817be00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f817c114870 con 0x7f817c0772f0 2024-04-28T13:06:44.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.925+0000 7f817be00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f817c114b70 con 0x7f817c078ed0 2024-04-28T13:06:44.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.925+0000 7f8173e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f817c078530 0x7f817c19f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:44.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.925+0000 7f8173e00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f817c078530 0x7f817c19f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:46412/0 (socket says 172.21.15.84:46412) 2024-04-28T13:06:44.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.925+0000 7f8173e00640 1 -- 172.21.15.84:0/231048104 learned_addr learned my addr 172.21.15.84:0/231048104 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:44.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.925+0000 7f817b400640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f817c078ed0 0x7f817c1a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:44.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.925+0000 7f817aa00640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f817c0772f0 0x7f817c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:44.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.926+0000 7f8173e00640 1 -- 172.21.15.84:0/231048104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f817c078ed0 msgr2=0x7f817c1a6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:44.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.926+0000 7f8173e00640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f817c078ed0 0x7f817c1a6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:44.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.926+0000 7f8173e00640 1 -- 172.21.15.84:0/231048104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f817c0772f0 msgr2=0x7f817c19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:44.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.926+0000 7f8173e00640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f817c0772f0 0x7f817c19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:44.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.926+0000 7f8173e00640 1 -- 172.21.15.84:0/231048104 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c1a6c90 con 0x7f817c078530 2024-04-28T13:06:44.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.926+0000 7f8173e00640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f817c078530 0x7f817c19f5b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f816400e9f0 tx=0x7f816400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:44.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.926+0000 7f8173400640 1 -- 172.21.15.84:0/231048104 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816400cd80 con 0x7f817c078530 2024-04-28T13:06:44.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.926+0000 7f8173400640 1 -- 172.21.15.84:0/231048104 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81640054d0 con 0x7f817c078530 2024-04-28T13:06:44.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.926+0000 7f817be00640 1 -- 172.21.15.84:0/231048104 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f817c1a6f80 con 0x7f817c078530 2024-04-28T13:06:44.931 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.926+0000 7f8173400640 1 -- 172.21.15.84:0/231048104 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8164010690 con 0x7f817c078530 2024-04-28T13:06:44.931 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.926+0000 7f817be00640 1 -- 172.21.15.84:0/231048104 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f817c1a73f0 con 0x7f817c078530 2024-04-28T13:06:44.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.927+0000 7f817be00640 1 -- 172.21.15.84:0/231048104 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f817c1113e0 con 0x7f817c078530 2024-04-28T13:06:44.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.930+0000 7f8173400640 1 -- 172.21.15.84:0/231048104 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8164004e50 con 0x7f817c078530 2024-04-28T13:06:44.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.931+0000 7f8173400640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f81600a3c40 0x7f81600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:44.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.931+0000 7f8173400640 1 -- 172.21.15.84:0/231048104 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8883+0+0 (secure 0 0 0) 0x7f81640d0df0 con 0x7f817c078530 2024-04-28T13:06:44.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.931+0000 7f817aa00640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f81600a3c40 0x7f81600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:44.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.932+0000 7f817aa00640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f81600a3c40 0x7f81600a6100 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f816c00bbe0 tx=0x7f816c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:44.934 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:44.934+0000 7f8173400640 1 -- 172.21.15.84:0/231048104 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8164054060 con 0x7f817c078530 2024-04-28T13:06:44.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:44 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3796225270' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:06:44.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:44 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:06:45.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.136+0000 7f817be00640 1 -- 172.21.15.84:0/231048104 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f817c02dad0 con 0x7f817c078530 2024-04-28T13:06:45.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:44 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3796225270' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:06:45.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:44 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:06:45.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:44 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3796225270' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:06:45.171 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:44 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:06:45.712 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.711+0000 7f8173400640 1 -- 172.21.15.84:0/231048104 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v97) ==== 169+0+0 (secure 0 0 0) 0x7f816409d380 con 0x7f817c078530 2024-04-28T13:06:45.712 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'cephfs_data' removed 2024-04-28T13:06:45.715 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.715+0000 7f817be00640 1 -- 172.21.15.84:0/231048104 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f81600a3c40 msgr2=0x7f81600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:45.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.715+0000 7f817be00640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f81600a3c40 0x7f81600a6100 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f816c00bbe0 tx=0x7f816c00b390 comp rx=0 tx=0).stop 2024-04-28T13:06:45.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.715+0000 7f817be00640 1 -- 172.21.15.84:0/231048104 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f817c078530 msgr2=0x7f817c19f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:45.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.715+0000 7f817be00640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f817c078530 0x7f817c19f5b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f816400e9f0 tx=0x7f816400edf0 comp rx=0 tx=0).stop 2024-04-28T13:06:45.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.715+0000 7f817be00640 1 -- 172.21.15.84:0/231048104 shutdown_connections 2024-04-28T13:06:45.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.715+0000 7f817be00640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f81600a3c40 0x7f81600a6100 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:45.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.715+0000 7f817be00640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f817c078ed0 0x7f817c1a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:45.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.715+0000 7f817be00640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f817c078530 0x7f817c19f5b0 secure :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f816400e9f0 tx=0x7f816400edf0 comp rx=0 tx=0).stop 2024-04-28T13:06:45.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.715+0000 7f817be00640 1 --2- 172.21.15.84:0/231048104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f817c0772f0 0x7f817c19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:45.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.715+0000 7f817be00640 1 -- 172.21.15.84:0/231048104 >> 172.21.15.84:0/231048104 conn(0x7f817c0fffe0 msgr2=0x7f817c101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:45.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.718+0000 7f817be00640 1 -- 172.21.15.84:0/231048104 shutdown_connections 2024-04-28T13:06:45.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.718+0000 7f817be00640 1 -- 172.21.15.84:0/231048104 wait complete. 2024-04-28T13:06:45.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-28T13:06:45.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.939+0000 7f83cbe00640 1 -- 172.21.15.84:0/2720658661 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f83cc109fa0 msgr2=0x7f83cc10a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:45.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.939+0000 7f83cbe00640 1 --2- 172.21.15.84:0/2720658661 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f83cc109fa0 0x7f83cc10a420 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009900 tx=0x7f83b802a410 comp rx=0 tx=0).stop 2024-04-28T13:06:45.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.939+0000 7f83cbe00640 1 -- 172.21.15.84:0/2720658661 shutdown_connections 2024-04-28T13:06:45.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.939+0000 7f83cbe00640 1 --2- 172.21.15.84:0/2720658661 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83cc10a960 0x7f83cc111280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:45.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.939+0000 7f83cbe00640 1 --2- 172.21.15.84:0/2720658661 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f83cc109fa0 0x7f83cc10a420 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:45.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.939+0000 7f83cbe00640 1 --2- 172.21.15.84:0/2720658661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83cc108e50 0x7f83cc109270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:45.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.939+0000 7f83cbe00640 1 -- 172.21.15.84:0/2720658661 >> 172.21.15.84:0/2720658661 conn(0x7f83cc077de0 msgr2=0x7f83cc07a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:45.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.939+0000 7f83cbe00640 1 -- 172.21.15.84:0/2720658661 shutdown_connections 2024-04-28T13:06:45.941 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.939+0000 7f83cbe00640 1 -- 172.21.15.84:0/2720658661 wait complete. 2024-04-28T13:06:45.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.940+0000 7f83cbe00640 1 Processor -- start 2024-04-28T13:06:45.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.940+0000 7f83cbe00640 1 -- start start 2024-04-28T13:06:45.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.941+0000 7f83cbe00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f83cc108e50 0x7f83cc19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:45.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.941+0000 7f83cbe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83cc109fa0 0x7f83cc19f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:45.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.941+0000 7f83cbe00640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83cc10a960 0x7f83cc1a6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:45.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.941+0000 7f83cbe00640 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc1149b0 con 0x7f83cc108e50 2024-04-28T13:06:45.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.941+0000 7f83cbe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc114830 con 0x7f83cc109fa0 2024-04-28T13:06:45.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.941+0000 7f83cbe00640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc114b30 con 0x7f83cc10a960 2024-04-28T13:06:45.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.941+0000 7f83caa00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f83cc108e50 0x7f83cc19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:45.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.941+0000 7f83caa00640 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f83cc108e50 0x7f83cc19f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:46438/0 (socket says 172.21.15.84:46438) 2024-04-28T13:06:45.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.941+0000 7f83caa00640 1 -- 172.21.15.84:0/3899731819 learned_addr learned my addr 172.21.15.84:0/3899731819 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-04-28T13:06:45.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.941+0000 7f83cb400640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83cc10a960 0x7f83cc1a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:45.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.941+0000 7f83ca000640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83cc109fa0 0x7f83cc19f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:45.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.942+0000 7f83caa00640 1 -- 172.21.15.84:0/3899731819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83cc10a960 msgr2=0x7f83cc1a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:45.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.942+0000 7f83caa00640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83cc10a960 0x7f83cc1a6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:45.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.942+0000 7f83caa00640 1 -- 172.21.15.84:0/3899731819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83cc109fa0 msgr2=0x7f83cc19f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:45.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.942+0000 7f83caa00640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83cc109fa0 0x7f83cc19f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:45.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.942+0000 7f83caa00640 1 -- 172.21.15.84:0/3899731819 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83cc1a6c70 con 0x7f83cc108e50 2024-04-28T13:06:45.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.942+0000 7f83caa00640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f83cc108e50 0x7f83cc19f050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00ea60 tx=0x7f83bc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:45.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.942+0000 7f83d0a00640 1 -- 172.21.15.84:0/3899731819 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc00cde0 con 0x7f83cc108e50 2024-04-28T13:06:45.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.942+0000 7f83d0a00640 1 -- 172.21.15.84:0/3899731819 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83bc0035f0 con 0x7f83cc108e50 2024-04-28T13:06:45.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.942+0000 7f83cbe00640 1 -- 172.21.15.84:0/3899731819 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83cc1a6f60 con 0x7f83cc108e50 2024-04-28T13:06:45.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.942+0000 7f83d0a00640 1 -- 172.21.15.84:0/3899731819 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc010690 con 0x7f83cc108e50 2024-04-28T13:06:45.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.942+0000 7f83cbe00640 1 -- 172.21.15.84:0/3899731819 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83cc1a73c0 con 0x7f83cc108e50 2024-04-28T13:06:45.947 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.944+0000 7f83cbe00640 1 -- 172.21.15.84:0/3899731819 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83cc110160 con 0x7f83cc108e50 2024-04-28T13:06:45.947 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.945+0000 7f83d0a00640 1 -- 172.21.15.84:0/3899731819 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f83bc0037a0 con 0x7f83cc108e50 2024-04-28T13:06:45.948 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.947+0000 7f83d0a00640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f83b00a3c40 0x7f83b00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-28T13:06:45.949 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.947+0000 7f83d0a00640 1 -- 172.21.15.84:0/3899731819 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8324+0+0 (secure 0 0 0) 0x7f83bc0d08d0 con 0x7f83cc108e50 2024-04-28T13:06:45.949 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.947+0000 7f83ca000640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f83b00a3c40 0x7f83b00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-28T13:06:45.949 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.948+0000 7f83ca000640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f83b00a3c40 0x7f83b00a6100 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009230 tx=0x7f83b80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-28T13:06:45.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:45.953+0000 7f83d0a00640 1 -- 172.21.15.84:0/3899731819 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f83bc09d090 con 0x7f83cc108e50 2024-04-28T13:06:45.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:45 smithi161 ceph-mon[62387]: pgmap v872: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-28T13:06:45.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:45 smithi161 ceph-mon[62387]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:06:45.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:45 smithi161 ceph-mon[62387]: osdmap e96: 12 total, 12 up, 12 in 2024-04-28T13:06:45.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:45 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/231048104' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:06:46.072 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:45 smithi084 ceph-mon[28081]: pgmap v872: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-28T13:06:46.073 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:45 smithi084 ceph-mon[28081]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:06:46.073 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:45 smithi084 ceph-mon[28081]: osdmap e96: 12 total, 12 up, 12 in 2024-04-28T13:06:46.073 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:45 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/231048104' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:06:46.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.156+0000 7f83cbe00640 1 -- 172.21.15.84:0/3899731819 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f83cc02dad0 con 0x7f83cc108e50 2024-04-28T13:06:46.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:45 smithi179 ceph-mon[63029]: pgmap v872: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-28T13:06:46.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:45 smithi179 ceph-mon[63029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:06:46.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:45 smithi179 ceph-mon[63029]: osdmap e96: 12 total, 12 up, 12 in 2024-04-28T13:06:46.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:45 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/231048104' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:06:46.722 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.721+0000 7f83d0a00640 1 -- 172.21.15.84:0/3899731819 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v98) ==== 178+0+0 (secure 0 0 0) 0x7f83bc09c990 con 0x7f83cc108e50 2024-04-28T13:06:46.723 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'cephfs_data_ec' removed 2024-04-28T13:06:46.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.726+0000 7f83cbe00640 1 -- 172.21.15.84:0/3899731819 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f83b00a3c40 msgr2=0x7f83b00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:46.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.726+0000 7f83cbe00640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f83b00a3c40 0x7f83b00a6100 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009230 tx=0x7f83b80314a0 comp rx=0 tx=0).stop 2024-04-28T13:06:46.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.727+0000 7f83cbe00640 1 -- 172.21.15.84:0/3899731819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f83cc108e50 msgr2=0x7f83cc19f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-28T13:06:46.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.727+0000 7f83cbe00640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f83cc108e50 0x7f83cc19f050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00ea60 tx=0x7f83bc00ef20 comp rx=0 tx=0).stop 2024-04-28T13:06:46.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.727+0000 7f83cbe00640 1 -- 172.21.15.84:0/3899731819 shutdown_connections 2024-04-28T13:06:46.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.727+0000 7f83cbe00640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.84:6800/2396287759,v1:172.21.15.84:6801/2396287759] conn(0x7f83b00a3c40 0x7f83b00a6100 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:46.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.727+0000 7f83cbe00640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f83cc10a960 0x7f83cc1a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:46.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.727+0000 7f83cbe00640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83cc109fa0 0x7f83cc19f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:46.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.727+0000 7f83cbe00640 1 --2- 172.21.15.84:0/3899731819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f83cc108e50 0x7f83cc19f050 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-28T13:06:46.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.727+0000 7f83cbe00640 1 -- 172.21.15.84:0/3899731819 >> 172.21.15.84:0/3899731819 conn(0x7f83cc077de0 msgr2=0x7f83cc10d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-28T13:06:46.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.728+0000 7f83cbe00640 1 -- 172.21.15.84:0/3899731819 shutdown_connections 2024-04-28T13:06:46.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-04-28T13:06:46.728+0000 7f83cbe00640 1 -- 172.21.15.84:0/3899731819 wait complete. 2024-04-28T13:06:46.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:06:46.827 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:06:46.859 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:06:46.892 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-28T13:06:46.892 DEBUG:teuthology.orchestra.run.smithi084:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-28T13:06:46.948 DEBUG:teuthology.orchestra.run.smithi161:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-28T13:06:46.966 DEBUG:teuthology.orchestra.run.smithi179:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-28T13:06:46.984 INFO:tasks.cephadm:Stopping all daemons... 2024-04-28T13:06:46.984 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-28T13:06:46.984 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.a 2024-04-28T13:06:46.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:46 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/231048104' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:06:46.996 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:46 smithi161 ceph-mon[62387]: osdmap e97: 12 total, 12 up, 12 in 2024-04-28T13:06:46.997 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:46 smithi161 ceph-mon[62387]: from='client.? 172.21.15.84:0/3899731819' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:06:47.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:46 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/231048104' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:06:47.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:46 smithi084 ceph-mon[28081]: osdmap e97: 12 total, 12 up, 12 in 2024-04-28T13:06:47.058 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:46 smithi084 ceph-mon[28081]: from='client.? 172.21.15.84:0/3899731819' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:06:47.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:46 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/231048104' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-28T13:06:47.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:46 smithi179 ceph-mon[63029]: osdmap e97: 12 total, 12 up, 12 in 2024-04-28T13:06:47.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:46 smithi179 ceph-mon[63029]: from='client.? 172.21.15.84:0/3899731819' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-28T13:06:47.421 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:47 smithi084 systemd[1]: Stopping Ceph mon.a for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:06:47.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:47 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a[28077]: 2024-04-28T13:06:47.566+0000 7f089d400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:06:47.921 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:47 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a[28077]: 2024-04-28T13:06:47.566+0000 7f089d400700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-28T13:06:47.922 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:47 smithi084 podman[82291]: 2024-04-28 13:06:47.64127323 +0000 UTC m=+0.225304470 container died d77f816c8b1cc022e5e3fb02a1895259519bb9733db38fde60ffd0b21ccac878 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-28T13:06:48.674 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:48 smithi084 podman[82291]: 2024-04-28 13:06:48.252779782 +0000 UTC m=+0.836811066 container cleanup d77f816c8b1cc022e5e3fb02a1895259519bb9733db38fde60ffd0b21ccac878 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-28T13:06:48.674 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:48 smithi084 bash[82291]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a 2024-04-28T13:06:48.674 INFO:journalctl@ceph.mon.a.smithi084.stdout:Apr 28 13:06:48 smithi084 podman[82305]: 2024-04-28 13:06:48.496172974 +0000 UTC m=+0.841687724 container remove d77f816c8b1cc022e5e3fb02a1895259519bb9733db38fde60ffd0b21ccac878 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-28T13:06:49.076 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.a.service' 2024-04-28T13:06:49.202 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:49.202 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-28T13:06:49.202 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-28T13:06:49.203 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.b 2024-04-28T13:06:49.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:49 smithi161 systemd[1]: Stopping Ceph mon.b for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:06:49.826 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:49 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-b[62383]: 2024-04-28T13:06:49.520+0000 7f830b9bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:06:49.827 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:49 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-b[62383]: 2024-04-28T13:06:49.520+0000 7f830b9bb700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-28T13:06:49.827 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:49 smithi161 podman[96868]: 2024-04-28 13:06:49.572276673 +0000 UTC m=+0.172853875 container died 0411b65b5f435951d27b00c6bf7666905d59b29e34d9bfc83285fc8b6c093e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-b, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:06:50.236 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:50 smithi161 podman[96868]: 2024-04-28 13:06:50.010337498 +0000 UTC m=+0.610914701 container cleanup 0411b65b5f435951d27b00c6bf7666905d59b29e34d9bfc83285fc8b6c093e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:06:50.236 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:50 smithi161 bash[96868]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-b 2024-04-28T13:06:50.474 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.b.service' 2024-04-28T13:06:50.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:50 smithi161 podman[96882]: 2024-04-28 13:06:50.236825273 +0000 UTC m=+0.659023008 container remove 0411b65b5f435951d27b00c6bf7666905d59b29e34d9bfc83285fc8b6c093e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-28T13:06:50.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:50 smithi161 systemd[1]: ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-28T13:06:50.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:50 smithi161 systemd[1]: Stopped Ceph mon.b for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T13:06:50.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Apr 28 13:06:50 smithi161 systemd[1]: ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.b.service: Consumed 13.807s CPU time. 2024-04-28T13:06:50.519 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:50.520 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-28T13:06:50.520 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-28T13:06:50.520 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.c 2024-04-28T13:06:50.869 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:50 smithi179 systemd[1]: Stopping Ceph mon.c for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:06:51.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:50 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-c[63011]: 2024-04-28T13:06:50.868+0000 7f405bcb8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:06:51.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:50 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-c[63011]: 2024-04-28T13:06:50.868+0000 7f405bcb8700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-28T13:06:51.169 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:50 smithi179 podman[98443]: 2024-04-28 13:06:50.950356031 +0000 UTC m=+0.230622377 container died 39e2a04435a15370a1c8e93d533112bc7df7d3aee0100f20b9b58aead652f7d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-28T13:06:51.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:51 smithi179 podman[98443]: 2024-04-28 13:06:51.529717914 +0000 UTC m=+0.809984248 container cleanup 39e2a04435a15370a1c8e93d533112bc7df7d3aee0100f20b9b58aead652f7d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-28T13:06:51.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Apr 28 13:06:51 smithi179 bash[98443]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mon-c 2024-04-28T13:06:52.019 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mon.c.service' 2024-04-28T13:06:52.073 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:52.074 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-28T13:06:52.074 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-28T13:06:52.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.x 2024-04-28T13:06:52.421 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 13:06:52 smithi084 systemd[1]: Stopping Ceph mgr.x for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:06:52.921 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 13:06:52 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T13:06:52.638+0000 7f0210a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:06:52.922 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 13:06:52 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x[28308]: 2024-04-28T13:06:52.638+0000 7f0210a00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-28T13:06:52.922 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 13:06:52 smithi084 podman[82410]: 2024-04-28 13:06:52.731721141 +0000 UTC m=+0.238517223 container died 9ede3f8332e62dc302a924e39dd54ce9e7a4074dedaf96e7ccae3427f83ab661 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-28T13:06:53.421 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 13:06:53 smithi084 podman[82410]: 2024-04-28 13:06:53.257996703 +0000 UTC m=+0.764792795 container cleanup 9ede3f8332e62dc302a924e39dd54ce9e7a4074dedaf96e7ccae3427f83ab661 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-28T13:06:53.421 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 13:06:53 smithi084 bash[82410]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x 2024-04-28T13:06:53.752 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 13:06:53 smithi084 podman[82424]: 2024-04-28 13:06:53.501662263 +0000 UTC m=+0.759636656 container remove 9ede3f8332e62dc302a924e39dd54ce9e7a4074dedaf96e7ccae3427f83ab661 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:06:53.965 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-28T13:06:54.032 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 13:06:53 smithi084 systemd[1]: ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.x.service: Deactivated successfully. 2024-04-28T13:06:54.033 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 13:06:53 smithi084 systemd[1]: Stopped Ceph mgr.x for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T13:06:54.033 INFO:journalctl@ceph.mgr.x.smithi084.stdout:Apr 28 13:06:53 smithi084 systemd[1]: ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.x.service: Consumed 2min 35.360s CPU time. 2024-04-28T13:06:54.068 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:54.068 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-28T13:06:54.068 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-28T13:06:54.069 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.y 2024-04-28T13:06:54.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 13:06:54 smithi161 systemd[1]: Stopping Ceph mgr.y for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:06:54.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 13:06:54 smithi161 podman[96993]: 2024-04-28 13:06:54.402365191 +0000 UTC m=+0.139445257 container died 5cbde99f508802f5db752cf30af795ad1b90e31c87f349c76aaaa6678c69b24c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-28T13:06:54.996 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 13:06:54 smithi161 podman[96993]: 2024-04-28 13:06:54.765221653 +0000 UTC m=+0.502301708 container cleanup 5cbde99f508802f5db752cf30af795ad1b90e31c87f349c76aaaa6678c69b24c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-28T13:06:54.997 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Apr 28 13:06:54 smithi161 bash[96993]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-y 2024-04-28T13:06:55.170 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-28T13:06:55.204 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:55.204 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-28T13:06:55.204 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-28T13:06:55.205 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.z 2024-04-28T13:06:55.539 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 13:06:55 smithi179 systemd[1]: Stopping Ceph mgr.z for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:06:55.861 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 13:06:55 smithi179 podman[98580]: 2024-04-28 13:06:55.539627072 +0000 UTC m=+0.139045584 container died 5fab2ce76a48d607b8d43f78d686d3fd64d4ac335b2cecbdea2c6a86668a83b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-28T13:06:56.169 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 13:06:55 smithi179 podman[98580]: 2024-04-28 13:06:55.961307457 +0000 UTC m=+0.560725973 container cleanup 5fab2ce76a48d607b8d43f78d686d3fd64d4ac335b2cecbdea2c6a86668a83b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-28T13:06:56.169 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Apr 28 13:06:55 smithi179 bash[98580]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mgr-z 2024-04-28T13:06:56.442 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-28T13:06:56.476 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:06:56.476 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-28T13:06:56.477 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-28T13:06:56.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.0 2024-04-28T13:06:56.920 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:06:56 smithi084 systemd[1]: Stopping Ceph osd.0 for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:06:57.421 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:06:57 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-0[40337]: 2024-04-28T13:06:57.087+0000 7fee0e200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:06:57.422 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:06:57 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-0[40337]: 2024-04-28T13:06:57.087+0000 7fee0e200700 -1 osd.0 98 *** Got signal Terminated *** 2024-04-28T13:06:57.422 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:06:57 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-0[40337]: 2024-04-28T13:06:57.087+0000 7fee0e200700 -1 osd.0 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:07:02.868 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:07:02 smithi084 podman[82532]: 2024-04-28 13:07:02.505305286 +0000 UTC m=+5.585863023 container died 404118554f76036ae77c1d1c7152a1bcc4819446115745fc907c9919779351fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-28T13:07:03.171 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:07:02 smithi084 podman[82532]: 2024-04-28 13:07:02.987663844 +0000 UTC m=+6.068221604 container cleanup 404118554f76036ae77c1d1c7152a1bcc4819446115745fc907c9919779351fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-28T13:07:03.172 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:07:02 smithi084 bash[82532]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-0 2024-04-28T13:07:03.671 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:07:03 smithi084 podman[82552]: 2024-04-28 13:07:03.305942624 +0000 UTC m=+0.790891194 container remove 404118554f76036ae77c1d1c7152a1bcc4819446115745fc907c9919779351fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-28T13:07:04.203 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:07:03 smithi084 podman[82639]: 2024-04-28 13:07:03.857335208 +0000 UTC m=+0.033218086 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:07:04.203 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:07:04 smithi084 podman[82639]: 2024-04-28 13:07:04.032179007 +0000 UTC m=+0.208061910 container create 0621d85ad755a35529ec95dcc83265513b5e1d649865eee8d1ab73b9029ddc0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-28T13:07:04.672 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:07:04 smithi084 podman[82639]: 2024-04-28 13:07:04.338326622 +0000 UTC m=+0.514209529 container init 0621d85ad755a35529ec95dcc83265513b5e1d649865eee8d1ab73b9029ddc0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-04-28T13:07:04.672 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:07:04 smithi084 podman[82639]: 2024-04-28 13:07:04.349203457 +0000 UTC m=+0.525086363 container start 0621d85ad755a35529ec95dcc83265513b5e1d649865eee8d1ab73b9029ddc0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-28T13:07:04.672 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:07:04 smithi084 podman[82639]: 2024-04-28 13:07:04.418155212 +0000 UTC m=+0.594038118 container attach 0621d85ad755a35529ec95dcc83265513b5e1d649865eee8d1ab73b9029ddc0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-28T13:07:05.171 INFO:journalctl@ceph.osd.0.smithi084.stdout:Apr 28 13:07:04 smithi084 podman[82639]: 2024-04-28 13:07:04.815985766 +0000 UTC m=+0.991868696 container died 0621d85ad755a35529ec95dcc83265513b5e1d649865eee8d1ab73b9029ddc0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-28T13:07:05.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.0.service' 2024-04-28T13:07:05.776 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:05.776 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-28T13:07:05.776 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-28T13:07:05.776 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.1 2024-04-28T13:07:06.119 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:05 smithi161 systemd[1]: Stopping Ceph osd.1 for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:07:06.496 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:06 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1[66958]: 2024-04-28T13:07:06.117+0000 7f6c83054700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:06.497 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:06 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1[66958]: 2024-04-28T13:07:06.117+0000 7f6c83054700 -1 osd.1 98 *** Got signal Terminated *** 2024-04-28T13:07:06.497 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:06 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1[66958]: 2024-04-28T13:07:06.117+0000 7f6c83054700 -1 osd.1 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:07:11.977 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:11 smithi161 podman[97119]: 2024-04-28 13:07:11.669304368 +0000 UTC m=+5.690591834 container died 68af435f8076d3bc7fc9b6d038e26e174661f6dcbff3b1b0265cd0d9efe365fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:07:11.977 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:11 smithi161 podman[97119]: 2024-04-28 13:07:11.977511996 +0000 UTC m=+5.998799459 container cleanup 68af435f8076d3bc7fc9b6d038e26e174661f6dcbff3b1b0265cd0d9efe365fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-28T13:07:12.228 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:11 smithi161 bash[97119]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1 2024-04-28T13:07:12.496 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:12 smithi161 podman[97136]: 2024-04-28 13:07:12.228937631 +0000 UTC m=+0.556306734 container remove 68af435f8076d3bc7fc9b6d038e26e174661f6dcbff3b1b0265cd0d9efe365fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:07:12.849 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:12 smithi161 podman[97228]: 2024-04-28 13:07:12.49443623 +0000 UTC m=+0.017774468 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:07:12.849 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:12 smithi161 podman[97228]: 2024-04-28 13:07:12.596329169 +0000 UTC m=+0.119667394 container create fecef0c8cb1e781876e574ae911458e4a9700ba02a6f4a1560c1ae6d9a1df9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:07:13.145 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:12 smithi161 podman[97228]: 2024-04-28 13:07:12.849663341 +0000 UTC m=+0.373001562 container init fecef0c8cb1e781876e574ae911458e4a9700ba02a6f4a1560c1ae6d9a1df9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:07:13.145 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:12 smithi161 podman[97228]: 2024-04-28 13:07:12.853174473 +0000 UTC m=+0.376512697 container start fecef0c8cb1e781876e574ae911458e4a9700ba02a6f4a1560c1ae6d9a1df9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-28T13:07:13.145 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:12 smithi161 podman[97228]: 2024-04-28 13:07:12.940731916 +0000 UTC m=+0.464070152 container attach fecef0c8cb1e781876e574ae911458e4a9700ba02a6f4a1560c1ae6d9a1df9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-28T13:07:13.422 INFO:journalctl@ceph.osd.1.smithi161.stdout:Apr 28 13:07:13 smithi161 podman[97228]: 2024-04-28 13:07:13.14610903 +0000 UTC m=+0.669447255 container died fecef0c8cb1e781876e574ae911458e4a9700ba02a6f4a1560c1ae6d9a1df9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True) 2024-04-28T13:07:13.839 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.1.service' 2024-04-28T13:07:13.877 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:13.878 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-28T13:07:13.878 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-28T13:07:13.878 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.2 2024-04-28T13:07:14.169 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:13 smithi179 systemd[1]: Stopping Ceph osd.2 for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:07:14.669 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:14 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2[68161]: 2024-04-28T13:07:14.198+0000 7f6da0efa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:14.669 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:14 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2[68161]: 2024-04-28T13:07:14.198+0000 7f6da0efa700 -1 osd.2 98 *** Got signal Terminated *** 2024-04-28T13:07:14.669 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:14 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2[68161]: 2024-04-28T13:07:14.198+0000 7f6da0efa700 -1 osd.2 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:07:19.850 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:19 smithi179 podman[98705]: 2024-04-28 13:07:19.576898545 +0000 UTC m=+5.507800637 container died 6ba367a23292be89f6ff8b1add11c790376294e2a9f9dc61ac136a8ad5a9cc0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-28T13:07:20.169 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:19 smithi179 podman[98705]: 2024-04-28 13:07:19.949501036 +0000 UTC m=+5.880403119 container cleanup 6ba367a23292be89f6ff8b1add11c790376294e2a9f9dc61ac136a8ad5a9cc0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424) 2024-04-28T13:07:20.169 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:19 smithi179 bash[98705]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2 2024-04-28T13:07:20.568 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:20 smithi179 podman[98723]: 2024-04-28 13:07:20.234259489 +0000 UTC m=+0.652408762 container remove 6ba367a23292be89f6ff8b1add11c790376294e2a9f9dc61ac136a8ad5a9cc0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:07:20.880 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:20 smithi179 podman[98817]: 2024-04-28 13:07:20.471339734 +0000 UTC m=+0.017969311 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:07:20.880 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:20 smithi179 podman[98817]: 2024-04-28 13:07:20.610772962 +0000 UTC m=+0.157402540 container create 1f37e16e04809d8d0669eae62c1a46a6000bf61959f169a4e8fc3f6e103deded (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-28T13:07:20.881 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:20 smithi179 podman[98817]: 2024-04-28 13:07:20.881147133 +0000 UTC m=+0.427776718 container init 1f37e16e04809d8d0669eae62c1a46a6000bf61959f169a4e8fc3f6e103deded (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:07:21.169 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:20 smithi179 podman[98817]: 2024-04-28 13:07:20.884390659 +0000 UTC m=+0.431020242 container start 1f37e16e04809d8d0669eae62c1a46a6000bf61959f169a4e8fc3f6e103deded (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-28T13:07:21.169 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:20 smithi179 podman[98817]: 2024-04-28 13:07:20.972263119 +0000 UTC m=+0.518892703 container attach 1f37e16e04809d8d0669eae62c1a46a6000bf61959f169a4e8fc3f6e103deded (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-28T13:07:21.450 INFO:journalctl@ceph.osd.2.smithi179.stdout:Apr 28 13:07:21 smithi179 podman[98817]: 2024-04-28 13:07:21.175315934 +0000 UTC m=+0.721945512 container died 1f37e16e04809d8d0669eae62c1a46a6000bf61959f169a4e8fc3f6e103deded (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-28T13:07:21.779 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.2.service' 2024-04-28T13:07:21.820 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:21.820 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-28T13:07:21.821 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-28T13:07:21.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.3 2024-04-28T13:07:22.171 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:21 smithi084 systemd[1]: Stopping Ceph osd.3 for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:07:22.522 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:22 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3[46847]: 2024-04-28T13:07:22.420+0000 7f3627400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:22.523 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:22 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3[46847]: 2024-04-28T13:07:22.420+0000 7f3627400700 -1 osd.3 98 *** Got signal Terminated *** 2024-04-28T13:07:22.523 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:22 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3[46847]: 2024-04-28T13:07:22.420+0000 7f3627400700 -1 osd.3 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:07:28.295 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:27 smithi084 podman[82747]: 2024-04-28 13:07:27.951895345 +0000 UTC m=+5.693652677 container died 51cb1d4caafb59e0de55e95005784a02d057a4c022f84a68bf95c5e991f62238 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-04-28T13:07:28.625 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:28 smithi084 podman[82747]: 2024-04-28 13:07:28.40687532 +0000 UTC m=+6.148632675 container cleanup 51cb1d4caafb59e0de55e95005784a02d057a4c022f84a68bf95c5e991f62238 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-28T13:07:28.625 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:28 smithi084 bash[82747]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3 2024-04-28T13:07:28.920 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:28 smithi084 podman[82776]: 2024-04-28 13:07:28.624674349 +0000 UTC m=+0.652909753 container remove 51cb1d4caafb59e0de55e95005784a02d057a4c022f84a68bf95c5e991f62238 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:07:29.796 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:29 smithi084 podman[82862]: 2024-04-28 13:07:29.345344747 +0000 UTC m=+0.034405759 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:07:29.797 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:29 smithi084 podman[82862]: 2024-04-28 13:07:29.483959603 +0000 UTC m=+0.173020652 container create f1765ca6b9b614cf2fe8b6a53fc95651fae84e2b29646b9f28a6b63d6743eb16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:07:29.797 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:29 smithi084 podman[82862]: 2024-04-28 13:07:29.796339423 +0000 UTC m=+0.485400482 container init f1765ca6b9b614cf2fe8b6a53fc95651fae84e2b29646b9f28a6b63d6743eb16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-04-28T13:07:30.171 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:29 smithi084 podman[82862]: 2024-04-28 13:07:29.808025207 +0000 UTC m=+0.497086236 container start f1765ca6b9b614cf2fe8b6a53fc95651fae84e2b29646b9f28a6b63d6743eb16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-28T13:07:30.171 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:29 smithi084 podman[82862]: 2024-04-28 13:07:29.945203921 +0000 UTC m=+0.634265023 container attach f1765ca6b9b614cf2fe8b6a53fc95651fae84e2b29646b9f28a6b63d6743eb16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-28T13:07:30.617 INFO:journalctl@ceph.osd.3.smithi084.stdout:Apr 28 13:07:30 smithi084 podman[82862]: 2024-04-28 13:07:30.272189228 +0000 UTC m=+0.961250286 container died f1765ca6b9b614cf2fe8b6a53fc95651fae84e2b29646b9f28a6b63d6743eb16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-28T13:07:31.046 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.3.service' 2024-04-28T13:07:31.156 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:31.156 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-28T13:07:31.156 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-28T13:07:31.157 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.4 2024-04-28T13:07:31.496 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:31 smithi161 systemd[1]: Stopping Ceph osd.4 for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:07:31.997 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:31 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4[73486]: 2024-04-28T13:07:31.547+0000 7fd13dca0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:31.997 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:31 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4[73486]: 2024-04-28T13:07:31.547+0000 7fd13dca0700 -1 osd.4 98 *** Got signal Terminated *** 2024-04-28T13:07:31.997 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:31 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4[73486]: 2024-04-28T13:07:31.547+0000 7fd13dca0700 -1 osd.4 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:07:37.356 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:37 smithi161 podman[97345]: 2024-04-28 13:07:37.068166106 +0000 UTC m=+5.714931942 container died f14b268e5d2c4a18bb58f5f79a3f94a6164631c0df3f97ad04a141694aac390a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-04-28T13:07:37.714 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:37 smithi161 podman[97345]: 2024-04-28 13:07:37.455035263 +0000 UTC m=+6.101801108 container cleanup f14b268e5d2c4a18bb58f5f79a3f94a6164631c0df3f97ad04a141694aac390a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-28T13:07:37.714 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:37 smithi161 bash[97345]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4 2024-04-28T13:07:37.997 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:37 smithi161 podman[97362]: 2024-04-28 13:07:37.714574103 +0000 UTC m=+0.640559889 container remove f14b268e5d2c4a18bb58f5f79a3f94a6164631c0df3f97ad04a141694aac390a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-28T13:07:38.350 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:38 smithi161 podman[97451]: 2024-04-28 13:07:37.976067718 +0000 UTC m=+0.018108526 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:07:38.350 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:38 smithi161 podman[97451]: 2024-04-28 13:07:38.090309977 +0000 UTC m=+0.132350780 container create 52822b5fa76510ddb46ef292fcb38a9afbd61d50fc170a970eae459b41fe283a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-28T13:07:38.649 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:38 smithi161 podman[97451]: 2024-04-28 13:07:38.350893659 +0000 UTC m=+0.392934459 container init 52822b5fa76510ddb46ef292fcb38a9afbd61d50fc170a970eae459b41fe283a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-28T13:07:38.650 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:38 smithi161 podman[97451]: 2024-04-28 13:07:38.354109756 +0000 UTC m=+0.396150559 container start 52822b5fa76510ddb46ef292fcb38a9afbd61d50fc170a970eae459b41fe283a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-28T13:07:38.650 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:38 smithi161 podman[97451]: 2024-04-28 13:07:38.459318468 +0000 UTC m=+0.501359284 container attach 52822b5fa76510ddb46ef292fcb38a9afbd61d50fc170a970eae459b41fe283a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-28T13:07:38.961 INFO:journalctl@ceph.osd.4.smithi161.stdout:Apr 28 13:07:38 smithi161 podman[97451]: 2024-04-28 13:07:38.650724525 +0000 UTC m=+0.692765328 container died 52822b5fa76510ddb46ef292fcb38a9afbd61d50fc170a970eae459b41fe283a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:07:39.317 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.4.service' 2024-04-28T13:07:39.359 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:39.360 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-28T13:07:39.360 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-28T13:07:39.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.5 2024-04-28T13:07:39.669 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:39 smithi179 systemd[1]: Stopping Ceph osd.5 for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:07:40.169 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:39 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5[74676]: 2024-04-28T13:07:39.676+0000 7faef4f2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:40.169 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:39 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5[74676]: 2024-04-28T13:07:39.676+0000 7faef4f2d700 -1 osd.5 98 *** Got signal Terminated *** 2024-04-28T13:07:40.169 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:39 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5[74676]: 2024-04-28T13:07:39.676+0000 7faef4f2d700 -1 osd.5 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:07:45.406 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:45 smithi179 podman[98922]: 2024-04-28 13:07:45.139470613 +0000 UTC m=+5.582658980 container died 7b2827a0ae7b8fa3739fdca497fa9e3df10b8f781202637370296e53c09e9447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-28T13:07:45.669 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:45 smithi179 podman[98922]: 2024-04-28 13:07:45.505535238 +0000 UTC m=+5.948723605 container cleanup 7b2827a0ae7b8fa3739fdca497fa9e3df10b8f781202637370296e53c09e9447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-28T13:07:45.669 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:45 smithi179 bash[98922]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5 2024-04-28T13:07:46.169 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:45 smithi179 podman[98940]: 2024-04-28 13:07:45.823582807 +0000 UTC m=+0.678798710 container remove 7b2827a0ae7b8fa3739fdca497fa9e3df10b8f781202637370296e53c09e9447 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-28T13:07:46.485 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:46 smithi179 podman[99031]: 2024-04-28 13:07:46.083228439 +0000 UTC m=+0.018247371 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:07:46.485 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:46 smithi179 podman[99031]: 2024-04-28 13:07:46.208002942 +0000 UTC m=+0.143021869 container create 48251b8111543bb98af905e875002b54d511177b3e206d3da2aa7e81af227a4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-28T13:07:46.485 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:46 smithi179 podman[99031]: 2024-04-28 13:07:46.485959399 +0000 UTC m=+0.420978324 container init 48251b8111543bb98af905e875002b54d511177b3e206d3da2aa7e81af227a4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-28T13:07:46.786 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:46 smithi179 podman[99031]: 2024-04-28 13:07:46.489057725 +0000 UTC m=+0.424076650 container start 48251b8111543bb98af905e875002b54d511177b3e206d3da2aa7e81af227a4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:07:46.786 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:46 smithi179 podman[99031]: 2024-04-28 13:07:46.577576424 +0000 UTC m=+0.512595349 container attach 48251b8111543bb98af905e875002b54d511177b3e206d3da2aa7e81af227a4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-28T13:07:47.088 INFO:journalctl@ceph.osd.5.smithi179.stdout:Apr 28 13:07:46 smithi179 podman[99031]: 2024-04-28 13:07:46.787035971 +0000 UTC m=+0.722054902 container died 48251b8111543bb98af905e875002b54d511177b3e206d3da2aa7e81af227a4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-28T13:07:47.488 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.5.service' 2024-04-28T13:07:47.525 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:47.525 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-28T13:07:47.526 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-28T13:07:47.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.6 2024-04-28T13:07:47.921 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:47 smithi084 systemd[1]: Stopping Ceph osd.6 for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:07:48.421 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:48 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6[53405]: 2024-04-28T13:07:48.080+0000 7f75eac00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:48.421 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:48 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6[53405]: 2024-04-28T13:07:48.080+0000 7f75eac00700 -1 osd.6 98 *** Got signal Terminated *** 2024-04-28T13:07:48.421 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:48 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6[53405]: 2024-04-28T13:07:48.080+0000 7f75eac00700 -1 osd.6 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:07:53.829 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:53 smithi084 podman[82967]: 2024-04-28 13:07:53.521179113 +0000 UTC m=+5.591985851 container died e74e5c1dc540fb4ff7693accad432b3d14ccd6c284eb922969c84a26f452cbdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-28T13:07:54.171 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:53 smithi084 podman[82967]: 2024-04-28 13:07:53.991148638 +0000 UTC m=+6.061955420 container cleanup e74e5c1dc540fb4ff7693accad432b3d14ccd6c284eb922969c84a26f452cbdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-28T13:07:54.171 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:53 smithi084 bash[82967]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6 2024-04-28T13:07:54.671 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:54 smithi084 podman[82987]: 2024-04-28 13:07:54.250726417 +0000 UTC m=+0.715549801 container remove e74e5c1dc540fb4ff7693accad432b3d14ccd6c284eb922969c84a26f452cbdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424) 2024-04-28T13:07:55.414 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:55 smithi084 podman[83077]: 2024-04-28 13:07:54.959081623 +0000 UTC m=+0.034559996 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:07:55.414 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:55 smithi084 podman[83077]: 2024-04-28 13:07:55.068006945 +0000 UTC m=+0.143485377 container create 60d0b0e30d1379ee48d345fff1fed2f1af0593e3cb940fc4b1ad34e5e89cc256 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-28T13:07:55.414 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:55 smithi084 podman[83077]: 2024-04-28 13:07:55.413383125 +0000 UTC m=+0.488861531 container init 60d0b0e30d1379ee48d345fff1fed2f1af0593e3cb940fc4b1ad34e5e89cc256 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-28T13:07:55.671 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:55 smithi084 podman[83077]: 2024-04-28 13:07:55.425542786 +0000 UTC m=+0.501021158 container start 60d0b0e30d1379ee48d345fff1fed2f1af0593e3cb940fc4b1ad34e5e89cc256 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-28T13:07:55.671 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:55 smithi084 podman[83077]: 2024-04-28 13:07:55.512070289 +0000 UTC m=+0.587548714 container attach 60d0b0e30d1379ee48d345fff1fed2f1af0593e3cb940fc4b1ad34e5e89cc256 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-28T13:07:56.171 INFO:journalctl@ceph.osd.6.smithi084.stdout:Apr 28 13:07:55 smithi084 podman[83077]: 2024-04-28 13:07:55.874687722 +0000 UTC m=+0.950166107 container died 60d0b0e30d1379ee48d345fff1fed2f1af0593e3cb940fc4b1ad34e5e89cc256 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-28T13:07:56.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.6.service' 2024-04-28T13:07:56.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:07:56.783 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-28T13:07:56.783 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-28T13:07:56.783 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.7 2024-04-28T13:07:57.089 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:07:56 smithi161 systemd[1]: Stopping Ceph osd.7 for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:07:57.497 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:07:57 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7[80463]: 2024-04-28T13:07:57.088+0000 7f20b55be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:07:57.497 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:07:57 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7[80463]: 2024-04-28T13:07:57.088+0000 7f20b55be700 -1 osd.7 98 *** Got signal Terminated *** 2024-04-28T13:07:57.497 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:07:57 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7[80463]: 2024-04-28T13:07:57.088+0000 7f20b55be700 -1 osd.7 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:08:02.789 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:02 smithi161 podman[97556]: 2024-04-28 13:08:02.523338376 +0000 UTC m=+5.541963330 container died 250a3bd2e622890b90e35a0fdb41e2002a2c4e315fc09666ac690cfc74ba0622 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:08:03.190 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:02 smithi161 podman[97556]: 2024-04-28 13:08:02.889081468 +0000 UTC m=+5.907706411 container cleanup 250a3bd2e622890b90e35a0fdb41e2002a2c4e315fc09666ac690cfc74ba0622 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-28T13:08:03.190 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:02 smithi161 bash[97556]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7 2024-04-28T13:08:03.496 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:03 smithi161 podman[97585]: 2024-04-28 13:08:03.190414806 +0000 UTC m=+0.656584920 container remove 250a3bd2e622890b90e35a0fdb41e2002a2c4e315fc09666ac690cfc74ba0622 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-28T13:08:03.776 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:03 smithi161 podman[97680]: 2024-04-28 13:08:03.414418365 +0000 UTC m=+0.018172594 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:08:03.776 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:03 smithi161 podman[97680]: 2024-04-28 13:08:03.541261364 +0000 UTC m=+0.145015626 container create f4502b24f427543d26517b3710f5fd2c23c3a930d8051a9c5ebb7147f2d83055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-28T13:08:04.069 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:03 smithi161 podman[97680]: 2024-04-28 13:08:03.776989812 +0000 UTC m=+0.380744028 container init f4502b24f427543d26517b3710f5fd2c23c3a930d8051a9c5ebb7147f2d83055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:08:04.069 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:03 smithi161 podman[97680]: 2024-04-28 13:08:03.780211712 +0000 UTC m=+0.383965929 container start f4502b24f427543d26517b3710f5fd2c23c3a930d8051a9c5ebb7147f2d83055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-28T13:08:04.069 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:03 smithi161 podman[97680]: 2024-04-28 13:08:03.851999025 +0000 UTC m=+0.455753246 container attach f4502b24f427543d26517b3710f5fd2c23c3a930d8051a9c5ebb7147f2d83055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-28T13:08:04.354 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:04 smithi161 podman[97680]: 2024-04-28 13:08:04.069936569 +0000 UTC m=+0.673690799 container died f4502b24f427543d26517b3710f5fd2c23c3a930d8051a9c5ebb7147f2d83055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-28T13:08:04.404 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:04 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:04.014+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:04.588 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.7.service' 2024-04-28T13:08:04.610 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:04 smithi161 podman[97706]: 2024-04-28 13:08:04.572073139 +0000 UTC m=+0.495218598 container remove f4502b24f427543d26517b3710f5fd2c23c3a930d8051a9c5ebb7147f2d83055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-04-28T13:08:04.610 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:04 smithi161 systemd[1]: ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.7.service: Deactivated successfully. 2024-04-28T13:08:04.610 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:04 smithi161 systemd[1]: Stopped Ceph osd.7 for ddee604a-055b-11ef-bc93-c7b262605968. 2024-04-28T13:08:04.610 INFO:journalctl@ceph.osd.7.smithi161.stdout:Apr 28 13:08:04 smithi161 systemd[1]: ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.7.service: Consumed 10.329s CPU time. 2024-04-28T13:08:04.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:04.625 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-28T13:08:04.625 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-28T13:08:04.625 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.8 2024-04-28T13:08:04.959 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:04 smithi179 systemd[1]: Stopping Ceph osd.8 for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:08:05.366 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:05 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:05.013+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:05.366 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:04 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8[81674]: 2024-04-28T13:08:04.958+0000 7fcc97b4c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:05.366 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:04 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8[81674]: 2024-04-28T13:08:04.958+0000 7fcc97b4c700 -1 osd.8 98 *** Got signal Terminated *** 2024-04-28T13:08:05.366 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:04 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8[81674]: 2024-04-28T13:08:04.958+0000 7fcc97b4c700 -1 osd.8 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:08:06.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:05 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:05.966+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:07.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:06 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:06.925+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:08.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:07 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:07.956+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:09.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:08 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:08.954+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:10.352 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:10 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:10.002+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:10.669 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:10 smithi179 podman[99149]: 2024-04-28 13:08:10.353051953 +0000 UTC m=+5.516093628 container died 0ffb20db420174d90896c6ad18fa7241c3c7023e919b05321f0e4a9b67baa75c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-28T13:08:11.047 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:10 smithi179 podman[99149]: 2024-04-28 13:08:10.69248093 +0000 UTC m=+5.855522605 container cleanup 0ffb20db420174d90896c6ad18fa7241c3c7023e919b05321f0e4a9b67baa75c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-28T13:08:11.047 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:10 smithi179 bash[99149]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8 2024-04-28T13:08:11.048 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:10 smithi179 podman[99168]: 2024-04-28 13:08:10.935377511 +0000 UTC m=+0.580929684 container remove 0ffb20db420174d90896c6ad18fa7241c3c7023e919b05321f0e4a9b67baa75c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-28T13:08:11.358 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:11 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:11.047+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:11.630 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:11 smithi179 podman[99258]: 2024-04-28 13:08:11.262952029 +0000 UTC m=+0.018245012 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:08:11.630 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:11 smithi179 podman[99258]: 2024-04-28 13:08:11.394622659 +0000 UTC m=+0.149915674 container create 8c31c8c9576e6aa44bf6e7dc4e3b7efef7ce9a82b1ab39fabe27e1f6d4e41a93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-28T13:08:11.630 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:11 smithi179 podman[99258]: 2024-04-28 13:08:11.630188434 +0000 UTC m=+0.385481407 container init 8c31c8c9576e6aa44bf6e7dc4e3b7efef7ce9a82b1ab39fabe27e1f6d4e41a93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux ) 2024-04-28T13:08:11.919 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:11 smithi179 podman[99258]: 2024-04-28 13:08:11.633413311 +0000 UTC m=+0.388706287 container start 8c31c8c9576e6aa44bf6e7dc4e3b7efef7ce9a82b1ab39fabe27e1f6d4e41a93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-28T13:08:11.919 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:11 smithi179 podman[99258]: 2024-04-28 13:08:11.738625735 +0000 UTC m=+0.493918718 container attach 8c31c8c9576e6aa44bf6e7dc4e3b7efef7ce9a82b1ab39fabe27e1f6d4e41a93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-28T13:08:12.283 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:12 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:12.055+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:12.283 INFO:journalctl@ceph.osd.8.smithi179.stdout:Apr 28 13:08:11 smithi179 podman[99258]: 2024-04-28 13:08:11.932074567 +0000 UTC m=+0.687367544 container died 8c31c8c9576e6aa44bf6e7dc4e3b7efef7ce9a82b1ab39fabe27e1f6d4e41a93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-28T13:08:12.624 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.8.service' 2024-04-28T13:08:12.663 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:12.663 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-28T13:08:12.663 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-28T13:08:12.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.9 2024-04-28T13:08:12.997 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:12 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:12.620+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:13.171 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:12 smithi084 systemd[1]: Stopping Ceph osd.9 for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:08:13.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:13 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:13.071+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:13.672 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:13 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9[59990]: 2024-04-28T13:08:13.272+0000 7fe2fb200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:13.672 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:13 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9[59990]: 2024-04-28T13:08:13.272+0000 7fe2fb200700 -1 osd.9 98 *** Got signal Terminated *** 2024-04-28T13:08:13.673 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:13 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9[59990]: 2024-04-28T13:08:13.272+0000 7fe2fb200700 -1 osd.9 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:08:13.673 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:13 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9[59990]: 2024-04-28T13:08:13.442+0000 7fe2f6a00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:08.740083+0000 front 2024-04-28T13:07:08.739866+0000 (oldest deadline 2024-04-28T13:08:12.839640+0000) 2024-04-28T13:08:13.996 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:13 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:13.609+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:14.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:14 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:14.081+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:14.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:14 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:14.081+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:14.921 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:14 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9[59990]: 2024-04-28T13:08:14.446+0000 7fe2f6a00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:08.740083+0000 front 2024-04-28T13:07:08.739866+0000 (oldest deadline 2024-04-28T13:08:12.839640+0000) 2024-04-28T13:08:14.997 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:14 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:14.631+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:15.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:15 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:15.088+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:15.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:15 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:15.088+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:15.921 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:15 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9[59990]: 2024-04-28T13:08:15.461+0000 7fe2f6a00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:08.740083+0000 front 2024-04-28T13:07:08.739866+0000 (oldest deadline 2024-04-28T13:08:12.839640+0000) 2024-04-28T13:08:15.996 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:15 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:15.590+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:16.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:16 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:16.076+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:16.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:16 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:16.076+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:16.921 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:16 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9[59990]: 2024-04-28T13:08:16.461+0000 7fe2f6a00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:08.740083+0000 front 2024-04-28T13:07:08.739866+0000 (oldest deadline 2024-04-28T13:08:12.839640+0000) 2024-04-28T13:08:16.996 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:16 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:16.636+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:17.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:17 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:17.090+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:17.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:17 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:17.090+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:17.921 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:17 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9[59990]: 2024-04-28T13:08:17.456+0000 7fe2f6a00700 -1 osd.9 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:08.740083+0000 front 2024-04-28T13:07:08.739866+0000 (oldest deadline 2024-04-28T13:08:12.839640+0000) 2024-04-28T13:08:17.996 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:17 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:17.638+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:18.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:18 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:18.105+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:18.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:18 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:18.105+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:18.921 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:18 smithi084 podman[83184]: 2024-04-28 13:08:18.621730172 +0000 UTC m=+5.522827254 container died 0ef79bcf44a03aacf561721c3e98650bfc607f5cd018ba2f4c953ed8eae69d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-28T13:08:18.996 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:18 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:18.591+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:19.308 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:19 smithi084 podman[83184]: 2024-04-28 13:08:19.08149454 +0000 UTC m=+5.982591599 container cleanup 0ef79bcf44a03aacf561721c3e98650bfc607f5cd018ba2f4c953ed8eae69d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:08:19.308 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:19 smithi084 bash[83184]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9 2024-04-28T13:08:19.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:19 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:19.079+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:19.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:19 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:19.079+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:19.671 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:19 smithi084 podman[83202]: 2024-04-28 13:08:19.307538874 +0000 UTC m=+0.668656229 container remove 0ef79bcf44a03aacf561721c3e98650bfc607f5cd018ba2f4c953ed8eae69d2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-28T13:08:19.996 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:19 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:19.628+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:20.370 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:20 smithi084 podman[83290]: 2024-04-28 13:08:20.013907402 +0000 UTC m=+0.036684332 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:08:20.370 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:20 smithi084 podman[83290]: 2024-04-28 13:08:20.192107103 +0000 UTC m=+0.214884050 container create a896f8f020f15c4c4e1408e3429e3ade369bcf531a61b798f14d47b6c74d5a89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-28T13:08:20.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:20 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:20.114+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:20.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:20 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:20.114+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:20.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:20 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:20.114+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:20.671 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:20 smithi084 podman[83290]: 2024-04-28 13:08:20.495610812 +0000 UTC m=+0.518387734 container init a896f8f020f15c4c4e1408e3429e3ade369bcf531a61b798f14d47b6c74d5a89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-28T13:08:20.671 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:20 smithi084 podman[83290]: 2024-04-28 13:08:20.506966551 +0000 UTC m=+0.529743466 container start a896f8f020f15c4c4e1408e3429e3ade369bcf531a61b798f14d47b6c74d5a89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True) 2024-04-28T13:08:20.672 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:20 smithi084 podman[83290]: 2024-04-28 13:08:20.5856876 +0000 UTC m=+0.608464560 container attach a896f8f020f15c4c4e1408e3429e3ade369bcf531a61b798f14d47b6c74d5a89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:08:20.996 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:20 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:20.616+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:21.014 INFO:journalctl@ceph.osd.9.smithi084.stdout:Apr 28 13:08:20 smithi084 podman[83290]: 2024-04-28 13:08:20.946366321 +0000 UTC m=+0.969143252 container died a896f8f020f15c4c4e1408e3429e3ade369bcf531a61b798f14d47b6c74d5a89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-28T13:08:21.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:21 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:21.101+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:21.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:21 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:21.101+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:21.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:21 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:21.101+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:21.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.9.service' 2024-04-28T13:08:21.753 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:21.753 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-28T13:08:21.753 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-28T13:08:21.754 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.10 2024-04-28T13:08:21.997 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:21 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:21.567+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:21.997 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:21 smithi161 systemd[1]: Stopping Ceph osd.10 for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:08:22.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:22 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:22.092+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:22.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:22 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:22.092+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:22.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:22 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:22.092+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:22.496 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:22 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:22.090+0000 7ffbc704f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:22.497 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:22 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:22.090+0000 7ffbc704f700 -1 osd.10 98 *** Got signal Terminated *** 2024-04-28T13:08:22.497 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:22 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:22.090+0000 7ffbc704f700 -1 osd.10 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:08:22.996 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:22 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:22.602+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:23.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:23 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:23.056+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:23.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:23 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:23.056+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:23.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:23 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:23.056+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:23.996 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:23 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:23.626+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:23.997 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:23 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:23.626+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:17.720263+0000 front 2024-04-28T13:07:17.720179+0000 (oldest deadline 2024-04-28T13:08:23.020254+0000) 2024-04-28T13:08:24.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:24 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:24.031+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:24.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:24 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:24.031+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:24.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:24 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:24.031+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:24.997 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:24 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:24.614+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:24.997 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:24 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:24.614+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:17.720263+0000 front 2024-04-28T13:07:17.720179+0000 (oldest deadline 2024-04-28T13:08:23.020254+0000) 2024-04-28T13:08:25.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:25 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:25.022+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:25.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:25 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:25.022+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:25.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:25 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:25.022+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:25.996 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:25 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:25.651+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:25.997 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:25 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:25.651+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:17.720263+0000 front 2024-04-28T13:07:17.720179+0000 (oldest deadline 2024-04-28T13:08:23.020254+0000) 2024-04-28T13:08:26.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:25 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:25.998+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:26.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:25 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:25.998+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:26.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:25 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:25.998+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:26.898 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:26 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:26.647+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.718904+0000 front 2024-04-28T13:07:07.718931+0000 (oldest deadline 2024-04-28T13:08:11.819155+0000) 2024-04-28T13:08:26.898 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:26 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10[87241]: 2024-04-28T13:08:26.647+0000 7ffbc3643700 -1 osd.10 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:17.720263+0000 front 2024-04-28T13:07:17.720179+0000 (oldest deadline 2024-04-28T13:08:23.020254+0000) 2024-04-28T13:08:27.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:27 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:27.027+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:27.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:27 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:27.027+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:27.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:27 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:27.027+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:27.747 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:27 smithi161 podman[97786]: 2024-04-28 13:08:27.434358807 +0000 UTC m=+5.488516115 container died 0029c0b09854c125f1f711c5b6abd0e4a1e744ed5164fd094b6b0a62649f8b8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:08:28.056 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:27 smithi161 podman[97786]: 2024-04-28 13:08:27.806598115 +0000 UTC m=+5.860755422 container cleanup 0029c0b09854c125f1f711c5b6abd0e4a1e744ed5164fd094b6b0a62649f8b8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:08:28.057 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:27 smithi161 bash[97786]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10 2024-04-28T13:08:28.390 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:28 smithi161 podman[97803]: 2024-04-28 13:08:28.057363168 +0000 UTC m=+0.616308752 container remove 0029c0b09854c125f1f711c5b6abd0e4a1e744ed5164fd094b6b0a62649f8b8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:08:28.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:28 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:28.040+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:28.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:28 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:28.040+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:28.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:28 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:28.040+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:28.667 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:28 smithi161 podman[97894]: 2024-04-28 13:08:28.391431063 +0000 UTC m=+0.114283337 container create 0682e61a9876cdee21a0f35d12f9fd9caf6331306dc9949643f7e9087f638b25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5) 2024-04-28T13:08:28.668 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:28 smithi161 podman[97894]: 2024-04-28 13:08:28.295424866 +0000 UTC m=+0.018277165 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:08:28.668 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:28 smithi161 podman[97894]: 2024-04-28 13:08:28.593381688 +0000 UTC m=+0.316233959 container init 0682e61a9876cdee21a0f35d12f9fd9caf6331306dc9949643f7e9087f638b25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-28T13:08:28.668 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:28 smithi161 podman[97894]: 2024-04-28 13:08:28.596851696 +0000 UTC m=+0.319703969 container start 0682e61a9876cdee21a0f35d12f9fd9caf6331306dc9949643f7e9087f638b25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-28T13:08:28.668 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:28 smithi161 podman[97894]: 2024-04-28 13:08:28.668341225 +0000 UTC m=+0.391193514 container attach 0682e61a9876cdee21a0f35d12f9fd9caf6331306dc9949643f7e9087f638b25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-28T13:08:28.997 INFO:journalctl@ceph.osd.10.smithi161.stdout:Apr 28 13:08:28 smithi161 podman[97894]: 2024-04-28 13:08:28.886171795 +0000 UTC m=+0.609024084 container died 0682e61a9876cdee21a0f35d12f9fd9caf6331306dc9949643f7e9087f638b25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-28T13:08:29.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:29 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:29.079+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:29.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:29 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:29.079+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:29.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:29 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:29.079+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:29.484 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.10.service' 2024-04-28T13:08:29.521 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:29.522 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-28T13:08:29.522 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-28T13:08:29.522 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.11 2024-04-28T13:08:29.836 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:29 smithi179 systemd[1]: Stopping Ceph osd.11 for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:08:30.169 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:29 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:29.836+0000 7f25d52ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:30.169 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:29 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:29.836+0000 7f25d52ca700 -1 osd.11 98 *** Got signal Terminated *** 2024-04-28T13:08:30.169 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:29 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:29.836+0000 7f25d52ca700 -1 osd.11 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-28T13:08:30.169 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:30 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:30.068+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:30.169 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:30 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:30.068+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:30.170 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:30 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:30.068+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:31.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:31 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:31.094+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:31.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:31 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:31.094+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:31.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:31 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:31.094+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:32.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:32 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:32.067+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:32.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:32 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:32.067+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:32.420 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:32 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:32.067+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:33.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:33 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:33.093+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:33.419 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:33 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:33.093+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:33.420 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:33 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:33.093+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:34.320 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:34 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:34.047+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.84:6806 osd.0 since back 2024-04-28T13:07:01.373092+0000 front 2024-04-28T13:07:01.373253+0000 (oldest deadline 2024-04-28T13:08:03.072864+0000) 2024-04-28T13:08:34.321 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:34 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:34.047+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.161:6804 osd.1 since back 2024-04-28T13:07:07.773853+0000 front 2024-04-28T13:07:07.773932+0000 (oldest deadline 2024-04-28T13:08:13.673888+0000) 2024-04-28T13:08:34.321 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:34 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11[88413]: 2024-04-28T13:08:34.047+0000 7f25d18be700 -1 osd.11 98 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-04-28T13:07:18.275534+0000 front 2024-04-28T13:07:18.275560+0000 (oldest deadline 2024-04-28T13:08:19.975698+0000) 2024-04-28T13:08:35.668 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:35 smithi179 podman[99364]: 2024-04-28 13:08:35.357273462 +0000 UTC m=+5.645738093 container died 20cddb7e2896c13075ce39f763e35882c65a52c5e035cad6d5575d8a19293ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-28T13:08:35.919 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:35 smithi179 podman[99364]: 2024-04-28 13:08:35.669233006 +0000 UTC m=+5.957697634 container cleanup 20cddb7e2896c13075ce39f763e35882c65a52c5e035cad6d5575d8a19293ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-28T13:08:35.919 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:35 smithi179 bash[99364]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11 2024-04-28T13:08:35.920 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:35 smithi179 podman[99382]: 2024-04-28 13:08:35.862027519 +0000 UTC m=+0.508214470 container remove 20cddb7e2896c13075ce39f763e35882c65a52c5e035cad6d5575d8a19293ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424) 2024-04-28T13:08:36.540 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:36 smithi179 podman[99474]: 2024-04-28 13:08:36.192194374 +0000 UTC m=+0.018169830 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-28T13:08:36.540 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:36 smithi179 podman[99474]: 2024-04-28 13:08:36.30458592 +0000 UTC m=+0.130561386 container create a26b762b759561be1b4f4902d660c3e38a3752887fec348dcc64ac44ed6b1441 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:08:36.540 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:36 smithi179 podman[99474]: 2024-04-28 13:08:36.540787597 +0000 UTC m=+0.366763025 container init a26b762b759561be1b4f4902d660c3e38a3752887fec348dcc64ac44ed6b1441 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-28T13:08:36.825 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:36 smithi179 podman[99474]: 2024-04-28 13:08:36.544473756 +0000 UTC m=+0.370449189 container start a26b762b759561be1b4f4902d660c3e38a3752887fec348dcc64ac44ed6b1441 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-28T13:08:36.825 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:36 smithi179 podman[99474]: 2024-04-28 13:08:36.631987361 +0000 UTC m=+0.457962805 container attach a26b762b759561be1b4f4902d660c3e38a3752887fec348dcc64ac44ed6b1441 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:08:37.125 INFO:journalctl@ceph.osd.11.smithi179.stdout:Apr 28 13:08:36 smithi179 podman[99474]: 2024-04-28 13:08:36.825546033 +0000 UTC m=+0.651521462 container died a26b762b759561be1b4f4902d660c3e38a3752887fec348dcc64ac44ed6b1441 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:08:37.403 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@osd.11.service' 2024-04-28T13:08:37.441 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:37.441 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-28T13:08:37.441 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-28T13:08:37.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.a 2024-04-28T13:08:37.921 INFO:journalctl@ceph.mds.a.smithi084.stdout:Apr 28 13:08:37 smithi084 systemd[1]: Stopping Ceph mds.a for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:08:38.421 INFO:journalctl@ceph.mds.a.smithi084.stdout:Apr 28 13:08:38 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-a[66513]: 2024-04-28T13:08:38.053+0000 7fb910400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:38.422 INFO:journalctl@ceph.mds.a.smithi084.stdout:Apr 28 13:08:38 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-a[66513]: 2024-04-28T13:08:38.053+0000 7fb910400700 -1 mds.a *** got signal Terminated *** 2024-04-28T13:08:42.020 INFO:journalctl@ceph.mds.a.smithi084.stdout:Apr 28 13:08:41 smithi084 podman[83411]: 2024-04-28 13:08:41.745941055 +0000 UTC m=+3.848900607 container died d05f67529e911c7f1110b15308ced1779469a728f5a5aa56601239b3260f4d77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:08:42.398 INFO:journalctl@ceph.mds.a.smithi084.stdout:Apr 28 13:08:42 smithi084 podman[83411]: 2024-04-28 13:08:42.205906399 +0000 UTC m=+4.308865979 container cleanup d05f67529e911c7f1110b15308ced1779469a728f5a5aa56601239b3260f4d77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-28T13:08:42.398 INFO:journalctl@ceph.mds.a.smithi084.stdout:Apr 28 13:08:42 smithi084 bash[83411]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-a 2024-04-28T13:08:42.671 INFO:journalctl@ceph.mds.a.smithi084.stdout:Apr 28 13:08:42 smithi084 podman[83425]: 2024-04-28 13:08:42.398337457 +0000 UTC m=+0.639695004 container remove d05f67529e911c7f1110b15308ced1779469a728f5a5aa56601239b3260f4d77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-28T13:08:42.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.a.service' 2024-04-28T13:08:42.910 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:42.911 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-28T13:08:42.911 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-28T13:08:42.911 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.d 2024-04-28T13:08:43.421 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 13:08:43 smithi084 systemd[1]: Stopping Ceph mds.d for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:08:43.921 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 13:08:43 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-d[65483]: 2024-04-28T13:08:43.422+0000 7f5a14e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:43.921 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 13:08:43 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-d[65483]: 2024-04-28T13:08:43.422+0000 7f5a14e00700 -1 mds.d *** got signal Terminated *** 2024-04-28T13:08:46.980 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 13:08:46 smithi084 podman[83531]: 2024-04-28 13:08:46.675472828 +0000 UTC m=+3.376312072 container died a1ee823846e83887fd1c06b05a883b6fa4281c2d430af607a52bbfc423145871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:08:47.421 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 13:08:47 smithi084 podman[83531]: 2024-04-28 13:08:47.100240305 +0000 UTC m=+3.801079569 container cleanup a1ee823846e83887fd1c06b05a883b6fa4281c2d430af607a52bbfc423145871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-28T13:08:47.421 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 13:08:47 smithi084 bash[83531]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-d 2024-04-28T13:08:47.921 INFO:journalctl@ceph.mds.d.smithi084.stdout:Apr 28 13:08:47 smithi084 podman[83545]: 2024-04-28 13:08:47.463051166 +0000 UTC m=+0.774840634 container remove a1ee823846e83887fd1c06b05a883b6fa4281c2d430af607a52bbfc423145871 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-28T13:08:47.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.d.service' 2024-04-28T13:08:48.082 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:48.082 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-28T13:08:48.082 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-28T13:08:48.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.g 2024-04-28T13:08:48.671 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 13:08:48 smithi084 systemd[1]: Stopping Ceph mds.g for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:08:49.171 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 13:08:48 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g[66128]: 2024-04-28T13:08:48.691+0000 7fb606000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:49.171 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 13:08:48 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g[66128]: 2024-04-28T13:08:48.691+0000 7fb606000700 -1 mds.g *** got signal Terminated *** 2024-04-28T13:08:50.936 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 13:08:50 smithi084 podman[83652]: 2024-04-28 13:08:50.674318305 +0000 UTC m=+2.116933812 container died 9db94e8bc7f0ace0af82a8e93b939b2e2a7d9cbf601fe719dc87b5785b11107b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-28T13:08:51.263 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 13:08:51 smithi084 podman[83652]: 2024-04-28 13:08:51.053772544 +0000 UTC m=+2.496388047 container cleanup 9db94e8bc7f0ace0af82a8e93b939b2e2a7d9cbf601fe719dc87b5785b11107b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:08:51.263 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 13:08:51 smithi084 bash[83652]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g 2024-04-28T13:08:51.671 INFO:journalctl@ceph.mds.g.smithi084.stdout:Apr 28 13:08:51 smithi084 podman[83665]: 2024-04-28 13:08:51.262864576 +0000 UTC m=+0.574220894 container remove 9db94e8bc7f0ace0af82a8e93b939b2e2a7d9cbf601fe719dc87b5785b11107b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-28T13:08:51.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.g.service' 2024-04-28T13:08:51.838 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:51.838 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-28T13:08:51.838 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-28T13:08:51.838 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.j 2024-04-28T13:08:52.171 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 13:08:51 smithi084 systemd[1]: Stopping Ceph mds.j for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:08:52.921 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 13:08:52 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-j[65140]: 2024-04-28T13:08:52.453+0000 7f5a3e000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:52.921 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 13:08:52 smithi084 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-j[65140]: 2024-04-28T13:08:52.453+0000 7f5a3e000700 -1 mds.j *** got signal Terminated *** 2024-04-28T13:08:54.922 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 13:08:54 smithi084 podman[83771]: 2024-04-28 13:08:54.675626501 +0000 UTC m=+2.397178562 container died 80c5ddf586152c5553058c80caba116619d8d81f6b08d315617f30fbb461caea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-28T13:08:55.316 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 13:08:55 smithi084 podman[83771]: 2024-04-28 13:08:55.089705175 +0000 UTC m=+2.811257226 container cleanup 80c5ddf586152c5553058c80caba116619d8d81f6b08d315617f30fbb461caea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-28T13:08:55.316 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 13:08:55 smithi084 bash[83771]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-j 2024-04-28T13:08:55.671 INFO:journalctl@ceph.mds.j.smithi084.stdout:Apr 28 13:08:55 smithi084 podman[83797]: 2024-04-28 13:08:55.315454063 +0000 UTC m=+0.622602462 container remove 80c5ddf586152c5553058c80caba116619d8d81f6b08d315617f30fbb461caea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux ) 2024-04-28T13:08:55.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.j.service' 2024-04-28T13:08:55.967 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:55.967 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-28T13:08:55.967 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-28T13:08:55.968 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.b 2024-04-28T13:08:56.247 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 13:08:56 smithi161 systemd[1]: Stopping Ceph mds.b for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:08:56.247 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 13:08:56 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b[92461]: 2024-04-28T13:08:56.237+0000 7fe13b687700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:56.247 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 13:08:56 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b[92461]: 2024-04-28T13:08:56.237+0000 7fe13b687700 -1 mds.b *** got signal Terminated *** 2024-04-28T13:08:58.929 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 13:08:58 smithi161 podman[98001]: 2024-04-28 13:08:58.631092945 +0000 UTC m=+2.472166382 container died 15ab11a559f9636c7a77aa6c623cc2dc8ee17e988ea1cc7f66a385ed913847d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-28T13:08:58.929 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 13:08:58 smithi161 podman[98001]: 2024-04-28 13:08:58.929521767 +0000 UTC m=+2.770595203 container cleanup 15ab11a559f9636c7a77aa6c623cc2dc8ee17e988ea1cc7f66a385ed913847d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424) 2024-04-28T13:08:59.204 INFO:journalctl@ceph.mds.b.smithi161.stdout:Apr 28 13:08:58 smithi161 bash[98001]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-b 2024-04-28T13:08:59.357 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.b.service' 2024-04-28T13:08:59.391 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:08:59.392 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-28T13:08:59.392 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-28T13:08:59.392 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.e 2024-04-28T13:08:59.746 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 13:08:59 smithi161 systemd[1]: Stopping Ceph mds.e for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:08:59.747 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 13:08:59 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-e[91844]: 2024-04-28T13:08:59.716+0000 7f0a9050e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:08:59.747 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 13:08:59 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-e[91844]: 2024-04-28T13:08:59.716+0000 7f0a9050e700 -1 mds.e *** got signal Terminated *** 2024-04-28T13:09:02.925 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 13:09:02 smithi161 podman[98136]: 2024-04-28 13:09:02.644263101 +0000 UTC m=+3.031794393 container died 3492e64bd3bb539baaef5e44b55daa3af6fc34d3b36aa0a956cc2a85a78e2ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-28T13:09:03.246 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 13:09:03 smithi161 podman[98136]: 2024-04-28 13:09:03.024615062 +0000 UTC m=+3.412146373 container cleanup 3492e64bd3bb539baaef5e44b55daa3af6fc34d3b36aa0a956cc2a85a78e2ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-28T13:09:03.247 INFO:journalctl@ceph.mds.e.smithi161.stdout:Apr 28 13:09:03 smithi161 bash[98136]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-e 2024-04-28T13:09:03.400 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.e.service' 2024-04-28T13:09:03.432 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:09:03.432 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-28T13:09:03.432 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-28T13:09:03.432 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.h 2024-04-28T13:09:03.997 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 13:09:03 smithi161 systemd[1]: Stopping Ceph mds.h for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:09:03.997 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 13:09:03 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h[92154]: 2024-04-28T13:09:03.752+0000 7f2796a1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:09:03.997 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 13:09:03 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h[92154]: 2024-04-28T13:09:03.752+0000 7f2796a1e700 -1 mds.h *** got signal Terminated *** 2024-04-28T13:09:06.894 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 13:09:06 smithi161 podman[98258]: 2024-04-28 13:09:06.642166429 +0000 UTC m=+2.987228735 container died 7c99db4ea81783aec7378c64932e0c642b5fbff25effbcb433dd69fa1ec3c153 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-28T13:09:07.246 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 13:09:06 smithi161 podman[98258]: 2024-04-28 13:09:06.95251326 +0000 UTC m=+3.297575567 container cleanup 7c99db4ea81783aec7378c64932e0c642b5fbff25effbcb433dd69fa1ec3c153 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:09:07.246 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 13:09:06 smithi161 bash[98258]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h 2024-04-28T13:09:07.246 INFO:journalctl@ceph.mds.h.smithi161.stdout:Apr 28 13:09:07 smithi161 podman[98272]: 2024-04-28 13:09:07.136612835 +0000 UTC m=+0.494145482 container remove 7c99db4ea81783aec7378c64932e0c642b5fbff25effbcb433dd69fa1ec3c153 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-28T13:09:07.260 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.h.service' 2024-04-28T13:09:07.291 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:09:07.291 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-28T13:09:07.292 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-28T13:09:07.292 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.k 2024-04-28T13:09:07.511 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 13:09:07 smithi161 systemd[1]: Stopping Ceph mds.k for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:09:07.997 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 13:09:07 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-k[91525]: 2024-04-28T13:09:07.597+0000 7f6ba4c07700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:09:07.997 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 13:09:07 smithi161 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-k[91525]: 2024-04-28T13:09:07.597+0000 7f6ba4c07700 -1 mds.k *** got signal Terminated *** 2024-04-28T13:09:10.981 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 13:09:10 smithi161 podman[98379]: 2024-04-28 13:09:10.695869568 +0000 UTC m=+3.188146177 container died 5dbf90dedeaeca2431c0a93bb0e56b4b5a089c03db545e30ec2e662309e802d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0) 2024-04-28T13:09:11.247 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 13:09:11 smithi161 podman[98379]: 2024-04-28 13:09:11.064015401 +0000 UTC m=+3.556292008 container cleanup 5dbf90dedeaeca2431c0a93bb0e56b4b5a089c03db545e30ec2e662309e802d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-28T13:09:11.247 INFO:journalctl@ceph.mds.k.smithi161.stdout:Apr 28 13:09:11 smithi161 bash[98379]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-k 2024-04-28T13:09:11.481 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.k.service' 2024-04-28T13:09:11.511 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:09:11.512 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-28T13:09:11.512 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-28T13:09:11.512 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.c 2024-04-28T13:09:11.800 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 13:09:11 smithi179 systemd[1]: Stopping Ceph mds.c for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:09:12.061 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 13:09:11 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-c[93721]: 2024-04-28T13:09:11.799+0000 7fb657fe9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:09:12.061 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 13:09:11 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-c[93721]: 2024-04-28T13:09:11.799+0000 7fb657fe9700 -1 mds.c *** got signal Terminated *** 2024-04-28T13:09:14.919 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 13:09:14 smithi179 podman[99581]: 2024-04-28 13:09:14.633285481 +0000 UTC m=+2.930037447 container died bd088b7c273ca273e564b2da6a821f6311b85245e148e140eec00eff0dc66776 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-28T13:09:15.393 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 13:09:15 smithi179 podman[99581]: 2024-04-28 13:09:15.100717323 +0000 UTC m=+3.397469289 container cleanup bd088b7c273ca273e564b2da6a821f6311b85245e148e140eec00eff0dc66776 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-28T13:09:15.393 INFO:journalctl@ceph.mds.c.smithi179.stdout:Apr 28 13:09:15 smithi179 bash[99581]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-c 2024-04-28T13:09:15.517 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.c.service' 2024-04-28T13:09:15.552 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:09:15.552 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-28T13:09:15.552 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-28T13:09:15.552 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.f 2024-04-28T13:09:15.669 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 13:09:15 smithi179 systemd[1]: Stopping Ceph mds.f for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:09:16.169 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 13:09:15 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-f[93414]: 2024-04-28T13:09:15.879+0000 7f360ab2b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:09:16.169 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 13:09:15 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-f[93414]: 2024-04-28T13:09:15.879+0000 7f360ab2b700 -1 mds.f *** got signal Terminated *** 2024-04-28T13:09:20.169 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 13:09:19 smithi179 podman[99714]: 2024-04-28 13:09:19.892208824 +0000 UTC m=+4.124020334 container died 0b9c806d90b45c7178b44d99fab87e18329fcf58d7d0673709898fea76e9711b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-28T13:09:20.669 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 13:09:20 smithi179 podman[99714]: 2024-04-28 13:09:20.45368703 +0000 UTC m=+4.685498546 container cleanup 0b9c806d90b45c7178b44d99fab87e18329fcf58d7d0673709898fea76e9711b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-28T13:09:20.669 INFO:journalctl@ceph.mds.f.smithi179.stdout:Apr 28 13:09:20 smithi179 bash[99714]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-f 2024-04-28T13:09:20.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.f.service' 2024-04-28T13:09:20.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:09:20.903 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-28T13:09:20.903 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-28T13:09:20.903 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.i 2024-04-28T13:09:21.169 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 13:09:20 smithi179 systemd[1]: Stopping Ceph mds.i for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:09:21.669 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 13:09:21 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-i[94029]: 2024-04-28T13:09:21.231+0000 7f3540b02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:09:21.669 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 13:09:21 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-i[94029]: 2024-04-28T13:09:21.231+0000 7f3540b02700 -1 mds.i *** got signal Terminated *** 2024-04-28T13:09:22.882 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 13:09:22 smithi179 podman[99834]: 2024-04-28 13:09:22.635650993 +0000 UTC m=+1.518049572 container died b20aa40ebd16506b626bdcd14961a8edc726077bc22f1552de3dc5b00fe07d0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-28T13:09:23.169 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 13:09:22 smithi179 podman[99834]: 2024-04-28 13:09:22.965525361 +0000 UTC m=+1.847923945 container cleanup b20aa40ebd16506b626bdcd14961a8edc726077bc22f1552de3dc5b00fe07d0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-28T13:09:23.169 INFO:journalctl@ceph.mds.i.smithi179.stdout:Apr 28 13:09:22 smithi179 bash[99834]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-i 2024-04-28T13:09:23.381 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.i.service' 2024-04-28T13:09:23.411 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:09:23.412 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-28T13:09:23.412 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-28T13:09:23.412 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.l 2024-04-28T13:09:23.727 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 13:09:23 smithi179 systemd[1]: Stopping Ceph mds.l for ddee604a-055b-11ef-bc93-c7b262605968... 2024-04-28T13:09:24.169 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 13:09:23 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-l[93102]: 2024-04-28T13:09:23.727+0000 7f4850c40700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-28T13:09:24.169 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 13:09:23 smithi179 ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-l[93102]: 2024-04-28T13:09:23.727+0000 7f4850c40700 -1 mds.l *** got signal Terminated *** 2024-04-28T13:09:27.919 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 13:09:27 smithi179 podman[99955]: 2024-04-28 13:09:27.653932724 +0000 UTC m=+4.027398812 container died 2c8bc769d5e5d2f8b320963a2804e3f1e5df21fb442aa607a00ae22129e37700 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=) 2024-04-28T13:09:28.286 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 13:09:28 smithi179 podman[99955]: 2024-04-28 13:09:28.035848701 +0000 UTC m=+4.409314761 container cleanup 2c8bc769d5e5d2f8b320963a2804e3f1e5df21fb442aa607a00ae22129e37700 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-04-28T13:09:28.286 INFO:journalctl@ceph.mds.l.smithi179.stdout:Apr 28 13:09:28 smithi179 bash[99955]: ceph-ddee604a-055b-11ef-bc93-c7b262605968-mds-l 2024-04-28T13:09:28.403 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ddee604a-055b-11ef-bc93-c7b262605968@mds.l.service' 2024-04-28T13:09:28.433 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-28T13:09:28.433 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-28T13:09:28.434 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ddee604a-055b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-28T13:09:28.749 INFO:teuthology.orchestra.run.smithi084.stdout:Deleting cluster with fsid: ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T13:09:33.680 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ddee604a-055b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-28T13:09:33.871 INFO:teuthology.orchestra.run.smithi161.stdout:Deleting cluster with fsid: ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T13:09:36.226 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ddee604a-055b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-28T13:09:36.416 INFO:teuthology.orchestra.run.smithi179.stdout:Deleting cluster with fsid: ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T13:09:38.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:09:38.851 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:09:38.882 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-28T13:09:38.908 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-28T13:09:38.909 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874/remote/smithi084/crash 2024-04-28T13:09:38.910 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/crash -- . 2024-04-28T13:09:39.016 INFO:teuthology.orchestra.run.smithi084.stderr:tar: /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-28T13:09:39.016 INFO:teuthology.orchestra.run.smithi084.stderr:tar: Error is not recoverable: exiting now 2024-04-28T13:09:39.021 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874/remote/smithi161/crash 2024-04-28T13:09:39.022 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/crash -- . 2024-04-28T13:09:39.060 INFO:teuthology.orchestra.run.smithi161.stderr:tar: /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-28T13:09:39.060 INFO:teuthology.orchestra.run.smithi161.stderr:tar: Error is not recoverable: exiting now 2024-04-28T13:09:39.062 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874/remote/smithi179/crash 2024-04-28T13:09:39.063 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/crash -- . 2024-04-28T13:09:39.105 INFO:teuthology.orchestra.run.smithi179.stderr:tar: /var/lib/ceph/ddee604a-055b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-28T13:09:39.105 INFO:teuthology.orchestra.run.smithi179.stderr:tar: Error is not recoverable: exiting now 2024-04-28T13:09:39.106 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-28T13:09:39.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-28T13:09:39.213 INFO:tasks.cephadm:Compressing logs... 2024-04-28T13:09:39.213 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-28T13:09:39.218 DEBUG:teuthology.orchestra.run.smithi161:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-28T13:09:39.221 DEBUG:teuthology.orchestra.run.smithi179:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-28T13:09:39.245 INFO:teuthology.orchestra.run.smithi179.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-28T13:09:39.246 INFO:teuthology.orchestra.run.smithi179.stderr:: No such file or directory 2024-04-28T13:09:39.246 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.log 2024-04-28T13:09:39.246 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-28T13:09:39.246 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-28T13:09:39.246 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-28T13:09:39.246 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-28T13:09:39.247 INFO:teuthology.orchestra.run.smithi179.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-28T13:09:39.247 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-28T13:09:39.247 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-28T13:09:39.247 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.5.log: /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-28T13:09:39.248 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-28T13:09:39.248 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mon.c.log: /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mgr.z.log: 89.2% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-28T13:09:39.248 INFO:teuthology.orchestra.run.smithi179.stderr: 90.4% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-28T13:09:39.249 INFO:teuthology.orchestra.run.smithi161.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-28T13:09:39.249 INFO:teuthology.orchestra.run.smithi161.stderr:: No such file or directory 2024-04-28T13:09:39.249 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.log 2024-04-28T13:09:39.249 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-28T13:09:39.250 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-28T13:09:39.250 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-28T13:09:39.250 INFO:teuthology.orchestra.run.smithi179.stderr: 94.5% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-28T13:09:39.256 INFO:teuthology.orchestra.run.smithi179.stderr: 93.3% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-28T13:09:39.257 INFO:teuthology.orchestra.run.smithi161.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-28T13:09:39.257 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-28T13:09:39.257 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.7.log: /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-28T13:09:39.257 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-28T13:09:39.257 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-28T13:09:39.257 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-28T13:09:39.258 INFO:teuthology.orchestra.run.smithi161.stderr: 90.4% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-28T13:09:39.258 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-28T13:09:39.258 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-28T13:09:39.258 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-28T13:09:39.258 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.4% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-28T13:09:39.258 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-28T13:09:39.258 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.e.log: 93.2% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-28T13:09:39.261 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-28T13:09:39.262 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-28T13:09:39.262 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-28T13:09:39.262 INFO:teuthology.orchestra.run.smithi179.stderr: 82.4% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.11.log: 2024-04-28T13:09:39.262 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-28T13:09:39.262 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-28T13:09:39.266 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-28T13:09:39.273 INFO:teuthology.orchestra.run.smithi084.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-28T13:09:39.274 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-28T13:09:39.274 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-volume.log: 91.0% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-28T13:09:39.275 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.log 2024-04-28T13:09:39.275 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-28T13:09:39.276 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-28T13:09:39.277 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-28T13:09:39.278 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-28T13:09:39.281 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-28T13:09:39.282 INFO:teuthology.orchestra.run.smithi084.stderr: 89.1% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-28T13:09:39.282 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-28T13:09:39.282 INFO:teuthology.orchestra.run.smithi084.stderr: 90.2% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-28T13:09:39.282 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-28T13:09:39.285 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.l.log: 94.5% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-28T13:09:39.286 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-28T13:09:39.287 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.0.log: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-28T13:09:39.287 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-28T13:09:39.288 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-28T13:09:39.288 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-28T13:09:39.288 INFO:teuthology.orchestra.run.smithi084.stderr: 94.6% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.j.log.gz/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.6.log: 2024-04-28T13:09:39.291 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.7% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-28T13:09:39.291 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-28T13:09:39.292 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-28T13:09:39.293 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.a.log: /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-28T13:09:39.299 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-28T13:09:39.299 INFO:teuthology.orchestra.run.smithi084.stderr: 94.5% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-28T13:09:39.299 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-28T13:09:39.304 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.4% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-28T13:09:39.309 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-28T13:09:39.310 INFO:teuthology.orchestra.run.smithi161.stderr: 94.1% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-28T13:09:39.311 INFO:teuthology.orchestra.run.smithi084.stderr: 94.5% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-28T13:09:39.318 INFO:teuthology.orchestra.run.smithi179.stderr: 94.6% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-28T13:09:39.319 INFO:teuthology.orchestra.run.smithi084.stderr: 90.9% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-28T13:09:39.328 INFO:teuthology.orchestra.run.smithi161.stderr: 94.6% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-28T13:09:39.344 INFO:teuthology.orchestra.run.smithi179.stderr: 94.5% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-28T13:09:39.345 INFO:teuthology.orchestra.run.smithi084.stderr: 94.0% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-28T13:09:39.456 INFO:teuthology.orchestra.run.smithi161.stderr: 93.8% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-28T13:09:39.651 INFO:teuthology.orchestra.run.smithi179.stderr: 92.0% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-28T13:09:39.653 INFO:teuthology.orchestra.run.smithi084.stderr: 90.5% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-28T13:09:39.741 INFO:teuthology.orchestra.run.smithi179.stderr: 92.7% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-28T13:09:39.759 INFO:teuthology.orchestra.run.smithi161.stderr: 92.3% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-28T13:09:39.783 INFO:teuthology.orchestra.run.smithi179.stderr: 92.5% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-28T13:09:39.971 INFO:teuthology.orchestra.run.smithi084.stderr: 92.5% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-28T13:09:40.091 INFO:teuthology.orchestra.run.smithi161.stderr: 92.1% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-28T13:09:40.108 INFO:teuthology.orchestra.run.smithi084.stderr: 92.4% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-28T13:09:40.278 INFO:teuthology.orchestra.run.smithi084.stderr: 90.3% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-28T13:09:40.306 INFO:teuthology.orchestra.run.smithi084.stderr: 93.1% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-28T13:09:40.323 INFO:teuthology.orchestra.run.smithi161.stderr: 93.1% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-28T13:09:40.338 INFO:teuthology.orchestra.run.smithi161.stderr: 93.0% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-28T13:09:40.376 INFO:teuthology.orchestra.run.smithi179.stderr: 93.1% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-28T13:09:40.392 INFO:teuthology.orchestra.run.smithi161.stderr: 93.2% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-28T13:09:40.394 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-28T13:09:40.394 INFO:teuthology.orchestra.run.smithi161.stderr:real 0m1.159s 2024-04-28T13:09:40.394 INFO:teuthology.orchestra.run.smithi161.stderr:user 0m4.825s 2024-04-28T13:09:40.394 INFO:teuthology.orchestra.run.smithi161.stderr:sys 0m0.201s 2024-04-28T13:09:40.549 INFO:teuthology.orchestra.run.smithi179.stderr: 92.8% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-28T13:09:40.551 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-04-28T13:09:40.551 INFO:teuthology.orchestra.run.smithi179.stderr:real 0m1.317s 2024-04-28T13:09:40.551 INFO:teuthology.orchestra.run.smithi179.stderr:user 0m3.909s 2024-04-28T13:09:40.551 INFO:teuthology.orchestra.run.smithi179.stderr:sys 0m0.153s 2024-04-28T13:09:40.639 INFO:teuthology.orchestra.run.smithi084.stderr: 92.8% -- replaced with /var/log/ceph/ddee604a-055b-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-28T13:09:40.643 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-04-28T13:09:40.643 INFO:teuthology.orchestra.run.smithi084.stderr:real 0m1.394s 2024-04-28T13:09:40.643 INFO:teuthology.orchestra.run.smithi084.stderr:user 0m4.999s 2024-04-28T13:09:40.643 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m0.444s 2024-04-28T13:09:40.645 INFO:tasks.cephadm:Archiving logs... 2024-04-28T13:09:40.645 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874/remote/smithi084/log 2024-04-28T13:09:40.646 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-28T13:09:41.444 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874/remote/smithi161/log 2024-04-28T13:09:41.445 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-28T13:09:42.153 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874/remote/smithi179/log 2024-04-28T13:09:42.154 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-28T13:09:42.748 INFO:tasks.cephadm:Removing cluster... 2024-04-28T13:09:42.748 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ddee604a-055b-11ef-bc93-c7b262605968 --force 2024-04-28T13:09:43.067 INFO:teuthology.orchestra.run.smithi084.stdout:Deleting cluster with fsid: ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T13:09:43.606 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ddee604a-055b-11ef-bc93-c7b262605968 --force 2024-04-28T13:09:43.798 INFO:teuthology.orchestra.run.smithi161.stdout:Deleting cluster with fsid: ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T13:09:44.048 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ddee604a-055b-11ef-bc93-c7b262605968 --force 2024-04-28T13:09:44.227 INFO:teuthology.orchestra.run.smithi179.stdout:Deleting cluster with fsid: ddee604a-055b-11ef-bc93-c7b262605968 2024-04-28T13:09:44.486 INFO:tasks.cephadm:Removing cephadm ... 2024-04-28T13:09:44.486 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-28T13:09:44.542 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-28T13:09:44.558 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-28T13:09:44.573 INFO:tasks.cephadm:Teardown complete 2024-04-28T13:09:44.574 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-28T13:09:44.584 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-28T13:09:44.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-28T13:09:44.591 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-28T13:09:44.601 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-28T13:09:44.747 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-28T13:09:44.758 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-28T13:09:44.758 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T13:09:44.766 DEBUG:teuthology.orchestra.run.smithi161:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T13:09:44.769 DEBUG:teuthology.orchestra.run.smithi179:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-28T13:09:44.780 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-04-28T13:09:44.782 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-04-28T13:09:44.786 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T13:09:44.786 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-04-28T13:09:44.786 INFO:teuthology.orchestra.run.smithi161.stdout:^* hv01.front.sepia.ceph.com 3 6 377 36 +1845us[+1848us] +/- 52ms 2024-04-28T13:09:44.786 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -1075us[-1075us] +/- 70ms 2024-04-28T13:09:44.786 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -1601us[-1601us] +/- 64ms 2024-04-28T13:09:44.786 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:09:44.787 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T13:09:44.788 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-04-28T13:09:44.788 INFO:teuthology.orchestra.run.smithi179.stdout:^* hv01.front.sepia.ceph.com 3 6 377 39 +1930us[+1894us] +/- 52ms 2024-04-28T13:09:44.788 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -975us[ -975us] +/- 70ms 2024-04-28T13:09:44.788 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 -1523us[-1523us] +/- 64ms 2024-04-28T13:09:44.788 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:09:44.811 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2024-04-28T13:09:44.820 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-28T13:09:44.820 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-04-28T13:09:44.820 INFO:teuthology.orchestra.run.smithi084.stdout:^* hv01.front.sepia.ceph.com 3 6 377 35 +1918us[+1913us] +/- 52ms 2024-04-28T13:09:44.820 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 35 -993us[ -993us] +/- 70ms 2024-04-28T13:09:44.820 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 -1503us[-1503us] +/- 64ms 2024-04-28T13:09:44.820 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-28T13:09:44.821 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-28T13:09:44.832 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-28T13:09:44.832 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-28T13:09:44.843 DEBUG:teuthology.orchestra.run.smithi084:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-28T13:09:45.057 DEBUG:teuthology.orchestra.run.smithi161:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-28T13:09:45.122 DEBUG:teuthology.orchestra.run.smithi179:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-28T13:09:45.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T13:09:45.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:09:45.281 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T13:09:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:09:45.307 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-28T13:09:45.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:09:45.334 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-28T13:09:45.344 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-28T13:09:45.353 INFO:teuthology.task.internal:Duration was 2612.313652 seconds 2024-04-28T13:09:45.354 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-28T13:09:45.363 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-28T13:09:45.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-28T13:09:45.372 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-28T13:09:45.374 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-28T13:09:45.409 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T13:09:45.410 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T13:09:45.498 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-28T13:09:45.752 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-28T13:09:45.752 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2024-04-28T13:09:45.752 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-28T13:09:45.813 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2024-04-28T13:09:45.813 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-28T13:09:45.837 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2024-04-28T13:09:45.838 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-28T13:09:45.863 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-28T13:09:45.863 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-28T13:09:45.870 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-28T13:09:45.880 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-28T13:09:45.928 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-28T13:09:45.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-28T13:09:45.933 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-28T13:09:45.945 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-28T13:09:46.422 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-28T13:09:46.434 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-28T13:09:46.434 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-28T13:09:46.497 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-28T13:09:46.521 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-28T13:09:46.546 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-28T13:09:46.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-28T13:09:46.565 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-28T13:09:46.567 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-28T13:09:46.587 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2024-04-28T13:09:46.608 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2024-04-28T13:09:46.646 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2024-04-28T13:09:46.687 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-28T13:09:46.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:09:46.732 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-28T13:09:46.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:09:46.746 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-28T13:09:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-28T13:09:46.760 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-28T13:09:46.770 INFO:teuthology.task.internal:Transferring archived files... 2024-04-28T13:09:46.771 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874/remote/smithi084 2024-04-28T13:09:46.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-28T13:09:46.875 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874/remote/smithi161 2024-04-28T13:09:46.875 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-28T13:09:46.910 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874/remote/smithi179 2024-04-28T13:09:46.910 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-28T13:09:46.946 INFO:teuthology.task.internal:Removing archive directory... 2024-04-28T13:09:46.946 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-28T13:09:46.954 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-28T13:09:46.956 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-28T13:09:47.001 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-28T13:09:47.013 INFO:teuthology.task.internal:Not uploading archives. 2024-04-28T13:09:47.014 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-28T13:09:47.024 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-28T13:09:47.025 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-28T13:09:47.032 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-28T13:09:47.035 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-28T13:09:47.046 INFO:teuthology.orchestra.run.smithi161.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 28 13:09 /home/ubuntu/cephtest 2024-04-28T13:09:47.049 INFO:teuthology.orchestra.run.smithi179.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 28 13:09 /home/ubuntu/cephtest 2024-04-28T13:09:47.081 INFO:teuthology.orchestra.run.smithi084.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 28 13:09 /home/ubuntu/cephtest 2024-04-28T13:09:47.084 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-28T13:09:47.095 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-28T13:09:47.118 INFO:teuthology.nuke:Checking targets against current locks 2024-04-28T13:09:47.145 DEBUG:teuthology.nuke:shortname: smithi084 2024-04-28T13:09:47.145 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-28T13:09:47.170 DEBUG:teuthology.nuke:shortname: smithi161 2024-04-28T13:09:47.170 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-28T13:09:47.176 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 12:12:13.133132', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGIgFwhf3vQnPVVn/zn96Zf863CehHmwn8t12xw3AiwW8ba02Swrvw+9m4vQ/QqLHcRw04F68ndZbDp1KGgvUj4='} 2024-04-28T13:09:47.194 DEBUG:teuthology.nuke:shortname: smithi179 2024-04-28T13:09:47.194 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-28T13:09:47.201 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 12:12:13.134531', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBCcIiiBvpOWfL0qT6g/oXGiHTkyqt88o58BkiWjrr94bU8jib7IsvHTiDcM9r2tYNj1WAnX0zVM5C+2gijo2mE='} 2024-04-28T13:09:47.213 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-28_11:41:23-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7676874', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-28 12:12:13.131415', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFOyXSh4ZYY5lu+FFO3Ced+cGQTgvUC82j/tH5f8Ne1HTKFH2vA8QREFAiCcMRtRaMtw4Q6j5ss4UxLXvg5qDtw='} 2024-04-28T13:09:47.227 INFO:teuthology.orchestra.console.smithi084:Power off 2024-04-28T13:09:47.228 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-28T13:09:47.253 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off 2024-04-28T13:09:47.254 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:09:47.272 INFO:teuthology.orchestra.console.smithi179:Power off 2024-04-28T13:09:47.273 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-28T13:09:47.288 INFO:teuthology.orchestra.console.smithi161:Power off 2024-04-28T13:09:47.289 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-28T13:09:47.297 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-04-28T13:09:47.297 DEBUG:teuthology.orchestra.console.smithi179:power off output: Chassis Power Control: Down/Off 2024-04-28T13:09:47.298 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:09:47.311 DEBUG:teuthology.orchestra.console.smithi161:power off output: Chassis Power Control: Down/Off 2024-04-28T13:09:47.312 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:09:47.322 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-04-28T13:09:47.334 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-04-28T13:09:51.299 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:09:51.322 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:09:51.335 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:09:51.410 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-04-28T13:09:51.432 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-04-28T13:09:51.445 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-04-28T13:09:55.415 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:09:55.433 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:09:55.445 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:09:55.526 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2024-04-28T13:09:55.543 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-04-28T13:09:55.555 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is off 2024-04-28T13:09:55.627 INFO:teuthology.orchestra.console.smithi084:Power off completed 2024-04-28T13:09:55.656 INFO:teuthology.orchestra.console.smithi161:Power off completed 2024-04-28T13:09:59.548 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-28T13:09:59.658 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is off 2024-04-28T13:09:59.759 INFO:teuthology.orchestra.console.smithi179:Power off completed 2024-04-28T13:09:59.865 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} duration: 2612.3136518001556 failure_reason: 'Command failed on smithi084 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs qa random 0.10''"' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=dc88b918d941469280c7aa2c8a1c6e81 status: fail success: false 2024-04-28T13:09:59.865 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-28T13:09:59.944 INFO:teuthology.run:FAIL